Role- MY | Senior Data Engineer

Job Description

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team The ideal candidate will have a strong background in designing building and maintaining scalable data pipelines and platforms On Premise and Cloud Ecosystem Microsoft Azure Preferrable with hands on expertise in modern data engineering tools and frameworks

Key Responsibilities

• Migrate data pipelines from existing data acquisition framework to the new GDP data acquisition framework
• Configure develop and deliver data ingestion scripts for loading data into T1 data layer
• Develop and manage ETLELT workflows ensuring data quality integrity and reliability
• Integrate and automate data quality checks and validation processes within data pipelines
• Deploy and manage containerized applications using Docker and orchestrate workloads on Kubernetes
• Work with modern data lake and warehouse technologies such as Iceberg
• Implement realtime data streaming solutions using Kafka
• Orchestrate complex workflows using Airflow
• Integrate with data catalog and governance tools such as Datahub and Ranger
• Collaborate with crossfunctional teams to understand business requirements and deliver data solutions
• Ensure security compliance and best practices in data management and governance

Key Skills Required

• Strong proficiency in Linux Python and Shell scripting
• Handson experience with Docker Kubernetes and container orchestration
• Handson experience with Minio and Azure Data Lake Storage ADLS using S3 protocols
• Experience with Apache Iceberg Kafka Airflow Datahub Trino and Ranger
• Proficiency in Java for data engineering tasks
• Solid understanding of data modeling data warehousing and big data technologies

Prior Experience

• Extensive background in building and maintaining data pipelinesETL processes
• Experience in implementing and integrating data quality frameworks
• Extensive experience in working on migration projects moving from legacy data pipelines to modernized tech stack
• Engagement Expectations
• Vendor resource will be expected to ramp up quickly and start contributing to active migration projects
• Must coordinate with internal leads product owners and platform teams to ensure project timelines are met
• Should be flexible to support multiple time zones if required
• Strong ownership mindset with commitment to quality and timely delivery"

Location Malaysia onsite

Contract for 1year and extendable