Senior Data Engineer

Role Summary 

 Design, develop, and maintain scalable data pipelines and data assets on Databricks and the Microsoft Azure Data Platform to support Business Intelligence and Advanced Analytics objectives.

• Lead the technical migration of legacy databases and on-premises data systems to a cloud-based lakehouse architecture using Delta Lake, including data modeling, transformation logic, and post-migration data validation.

Role and Responsibilities: 

 

• Design, develop, and maintain scalable data pipelines and data assets on Databricks and the Microsoft Azure Data Platform to support Business Intelligence and Advanced Analytics objectives.
• Lead the technical migration of legacy databases and on-premises data systems to a cloud-based lakehouse architecture using Delta Lake, including data modeling, transformation logic, and post-migration data validation.
• Develop distributed data processing applications using PySpark, Scala Spark, Python, and SQL for batch, incremental(CDC),and real-time streaming workloads.
• Build and customize data integration tools, data warehouses, data marts, and data lakes using Azure Data Factory, ADLS Gen2, and Kafka.
• Analyze business and system requirements and translate them into technical designs, data models, and software specifications aligned with enterprise architecture.
• Implement data governance, security, and data quality controls, including access management, lineage tracking, and automated data validation frameworks.
• Develop CI/CD pipelines and infrastructure-as-code for automated testing and deployment of data solutions using Databricks Asset Bundles and Git-based version control.
• Optimize performance, reliability, and cost of Spark workloads and cloud infrastructure through job tuning, cluster configuration, and storage optimization.
• Create tools and automation that help client engineering teams derive value from their data.
• Perform hands-on development, lead code reviews and testing, and establish engineering best practices across the team.
• Review solution approaches and provide technical guidance to onshore and offshore engineers to ensure delivery quality.
• Collaborate with client stakeholders to understand data needs and support timely delivery of data platform initiatives

Requirement/Competencies: 

• Bachelor’s or Master’s (preferred) degree in in a quantitative or technical field such as Computer Science, Information Technology, Computer Engineering or equivalent 

• 8+ years of experience comprising development for data systems at scale (building data warehouse, data marts, data lakes, ETL, analytic solutions, etc.) 

• Experience leading data migration / cloud transformation engagements 

• Experience with and understanding of Databricks, Azure Data Platform is a must.