DE&A - Core - Cloud Data Engineering - Sr. Azure Data Engineer

Role Overview

We are looking for an experienced Data Engineer with strong expertise in Azure Databricks, Azure Data Factory (ADF), and PySpark. The ideal candidate should have hands-on experience in building scalable data pipelines, integrating diverse data sources, and working on end-to-end data engineering solutions in the Azure ecosystem. This role involves working with structured, semi-structured, and unstructured data sources including SAP, SharePoint, JSON, XML, APIs, and files from Azure Blob Storage.

Key Responsibilities

Design, develop, and maintain data pipelines using Azure Databricks, ADF, and PySpark.

Ingest, transform, and process data from multiple sources including SAP, SharePoint, APIs, JSON, XML, and unstructured files.

• Implement efficient data ingestion from Azure Blob Storage and other Azure-native services.
• Collaborate with stakeholders to understand business requirements and translate them into scalable technical solutions.
• Optimize performance of PySpark scripts and Databricks workflows for large-scale data processing.
• Ensure data quality, governance, and security compliance throughout the pipeline lifecycle.
• Support integration of data into downstream systems for analytics and reporting.
• Troubleshoot data pipeline issues and provide timely resolutions.

Required Skills & Qualifications

3–7 years of hands-on experience in Data Engineering.

Strong Expertise In:

• Azure Databricks (PySpark, Delta Lake, notebooks, cluster management).
• Azure Data Factory (pipelines, triggers, linked services, monitoring).
• Azure Blob Storage (data ingestion, partitioning, schema management).
• Proficiency in PySpark for ETL/ELT development and large-scale data transformations.
• Experience working with SAP, SharePoint, APIs, JSON, XML, and unstructured files as data sources.
• Strong SQL skills for querying and transformation.
• Familiarity with data lake / data warehouse concepts.
• Understanding of CI/CD practices and version control (Git/Azure DevOps).

Excellent problem-solving and communication skills.

Good to Have (Optional)

• Exposure to Power BI or Synapse Analytics.
• Knowledge of data security, access management, and compliance in Azure.
• Experience with real-time/streaming ingestion (Kafka/Event Hub).

Why Join Us?

• Opportunity to work on cutting-edge Azure data projects.
• Exposure to diverse enterprise data sources including SAP and SharePoint.
• Career growth towards solution architect / lead roles.
• Collaborative and innovative work culture.