Job Overview

We are seeking an experienced Senior Data Engineer to join our dynamic team and drive our data engineering initiatives. The ideal candidate will have extensive experience with Azure cloud technologies and a proven track record of designing, implementing, and maintaining scalable data solutions. You will play a crucial role in building robust data pipelines, optimizing data workflows, and enabling data-driven decision making across the organization.

Key Responsibilities

• Design, develop, and maintain complex data pipelines using Azure Data Factory (ADF) and other Azure data services
• Build and optimize data processing workflows using Azure Databricks with Spark and PySpark
• Implement and manage Lake flow Declarative Pipelines for efficient data processing and transformation
• Develop and optimize SQL queries and stored procedures for data extraction, transformation, and loading (ETL) processes
• Create and maintain data models, schemas, and documentation to ensure data quality and consistency
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions
• Monitor and troubleshoot data pipeline performance, implementing improvements and optimizations as needed
• Ensure data security, privacy, and compliance with industry standards and regulations
• Mentor junior team members and provide technical guidance on data engineering best practices
• Stay current with emerging technologies and industry trends in data engineering and cloud computing
• Participate in code reviews, testing, and deployment processes to maintain high-quality deliverables
• Design and implement data governance frameworks and data quality monitoring solutions

Required Qualifications

• 8+ years of total professional experience in software development, data engineering, or related fields
• 7+ years of hands-on experience in data engineering roles with proven expertise in designing and implementing data solutions
• Extensive experience with Azure Data Factory (ADF) for building and orchestrating data pipelines
• Strong proficiency in Azure Databricks and Apache Spark for large-scale data processing
• Hands-on experience with Lake flow Declarative Pipelines and modern data pipeline architectures
• Advanced SQL skills with experience in query optimization and database design
• Proficient in PySpark and Python for data processing and transformation tasks
• Experience with Azure cloud services including Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics
• Strong understanding of data warehousing concepts, dimensional modeling, and ETL/ELT processes
• Knowledge of data governance, data quality, and data security best practices
• Experience with version control systems (Git) and CI/CD pipelines
• Excellent problem-solving skills and attention to detail
• Strong communication skills and ability to work effectively in cross-functional teams

Preferred Qualifications

• Bachelor's degree in Computer Science, Engineering, Mathematics, or related technical field
• Azure certifications such as Azure Data Engineer Associate (DP-203) or Azure Solutions Architect Expert
• Experience with additional Azure services like Azure Stream Analytics, Azure Event Hubs, or Azure Cosmos DB
• Knowledge of machine learning pipelines and MLOps practices
• Experience with Infrastructure as Code (IaC) tools like ARM templates or Terraform
• Familiarity with containerization technologies (Docker, Kubernetes)
• Experience with real-time data processing and streaming technologies
• Knowledge of other programming languages such as Scala or Java
• Experience with data visualization tools and business intelligence platforms
• Previous experience in mentoring and leading technical teams

Technical Skills

• Cloud Platforms: Microsoft Azure (primary focus)
• Data Processing: Azure Databricks, Apache Spark, PySpark
• Data Orchestration: Azure Data Factory (ADF), Lake flow Declarative Pipelines
• Databases: SQL Server, Azure SQL Database, Azure Synapse Analytics, Azure Data Lake
• Programming Languages: Python, SQL, PySpark
• Development Tools: Git, Azure DevOps, Visual Studio Code
• Data Formats: JSON, Parquet, Avro, CSV, XML
• Monitoring & Logging: Azure Monitor, Application Insights

What We Offer

• Competitive salary and comprehensive benefits package
• Opportunity to work with cutting-edge Azure technologies and data engineering tools
• Professional development opportunities and certification support
• Collaborative and innovative work environment
• Flexible work arrangements and work-life balance
• Health, dental, and vision insurance
• Retirement savings plan with company matching
• Paid time off and holidays
• Learning and development budget for continuous skill enhancement

Employment Type

Full-time, Permanent

Application Process

Interested candidates are encouraged to submit their resume along with a cover letter highlighting their relevant experience with Azure data engineering technologies. Please include specific examples of projects involving Azure Data Factory, Databricks, and data pipeline development.