Lead Azure Databricks Architect

Lead the design, development, and deployment of scalable data pipelines and workflows using core Azure data services such as Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage (ADLS), along with Azure Databricks.

Architect and optimize data solutions leveraging Medallion Architecture on Delta Lake using Azure Databricks as part of the overall Azure data platform strategy.

Develop and optimize ETL/ELT processes with Azure Data Factory and Azure Databricks to ensure efficient data ingestion, transformation, and delivery.

Collaborate closely with data scientists, analysts, and business stakeholders to translate business needs into effective Azure-based data engineering solutions.

Lead and mentor a team of data engineers, enforcing best practices in coding, testing, deployment, and documentation within the Azure environment.

Ensure data governance, quality, security, and compliance standards are maintained across Azure data pipelines.

Implement secure authentication and authorization mechanisms using Azure Active Directory (AAD) and manage secrets and keys securely using Azure Key Vault.

Implement CI/CD pipelines for automated deployment and monitoring using Azure DevOps.

Stay updated with Azure data services innovations and incorporate new technologies to enhance data infrastructure and team productivity.

8+ years of data engineering experience with a strong background working on the Azure data platform.

Solid hands-on experience with Azure Databricks, including Spark, PySpark, Delta Lake, and implementing Medallion Architecture.

Hands-on experience with Azure Data Factory for building scalable ETL/ELT pipelines.

Practical experience working with Azure Synapse Analytics and Azure Data Lake Storage (ADLS).

Strong skills in Python, SQL( T-SQL)  & PowerShell.

Experience with Azure Active Directory (AAD) integration for authentication and role-based access control.

Experience managing secrets and certificates with Azure Key Vault.

Experience with MLflow, Delta Live Tables, or Unity Catalog on Azure Databricks.

Familiarity with Azure Event Hubs, Azure Stream Analytics, or other Azure streaming services.

Knowledge of Azure Purview or other Azure-native data governance tools.

Hands-on experience with Snowflake, including data modeling, performance tuning, and query optimization in an Azure context.

Hands-on experience with Snowflake Streams and Tasks for incremental data processing.

Experience with Infrastructure as Code (IaC) on Azure using ARM templates or Terraform.