Azure Databricks Data Engineer

We are looking for an experienced Azure Databricks Data Engineer to help build and operate scalable, production-grade data platforms on Azure Databricks and Microsoft Azure.

The ideal candidate combines strong Spark engineering expertise with a production engineering mindset, treating data pipelines as reliable, enterprise-grade products rather than ad-hoc solutions. This role requires hands-on experience delivering secure, governed, high-performance batch and streaming data platforms at scale.

• Design, build, and maintain scalable batch and streaming pipelines using bronze/silver/gold architectures
• Develop reusable transformation frameworks and production-grade Spark workloads
• Implement and optimize Delta Lake solutions, including: ACID transactions, Partitioning strategies, Schema evolution, OPTIMIZE/ZORDER, Incremental and CDC processing patterns
• Configure and manage Unity Catalog permissions, governance, lineage, and auditability
• Perform Spark performance tuning and observability monitoring
• Integrate Databricks with Azure services including ADLS, ADF/Synapse, Key Vault, and Microsoft Entra ID
• Implement CI/CD pipelines and deployment automation using Azure DevOps or GitHub Actions
• Collaborate with architects, analysts, data scientists, and engineers to align technical implementation with business needs
• Continuously improve platform performance, reliability, cost efficiency, and standardization

• Strong hands-on expertise with Azure Databricks engineering

• Advanced Apache Spark experience using:

• PySpark or Scala

• Spark SQL

• Deep understanding of Delta Lake architecture and optimization techniques
• Experience building scalable enterprise data platforms and real-time pipelines
• Strong governance and security experience in multi-team environments
• Experience with service principals, managed identities, lineage, and auditability
• Strong troubleshooting and Spark performance optimization skills
• Azure integration experience across storage, orchestration, and security services
• Solid CI/CD and Infrastructure-as-Code exposure
• Comfortable working independently in consulting-style delivery environments