Associate Consultant - Data Engineer

• Data Pipeline ETL Development: Design and optimize ETL/ELT processes using Python/PySpark and SQL to efficiently ingest, transform, and integrate data from various sources into Azure data platforms. • Data Architecture Integration: Leverage Azure Data Factory, Databricks, and Synapse Analytics to establish a medallion architecture in ADLS, transforming raw data into refined and actionable formats. • Power BI Integration: Collaborate with power bi developers and data analysts to ensure seamless integration of user-facing elements. • Full Software Development Lifecycle: Participate in all phases of the SDLC, including requirement analysis, planning, development, testing, and deployment. • Cross-Functional Collaboration: Work closely with cross-functional teams, including data scientists, data analysts and product managers, to solve technical and business challenges collaboratively. • •Good to have: Experience in creating and managing data pipelines and notebooks with Microsoft Fabric. Familiarity with Gen AI techniques for enhancing data engineering tasks.

Roles & Responsibilities:

Mandatory Technical & Functional Skills

•Data Engineering Fundamentals: •Proficiency in SQL for querying and creating views, stored procedures, etc. •Proven experience in designing and implementing ETL/ELT pipelines for data ingestion, transformation, and integration across platforms. •Strong knowledge in handling large datasets and performing data analytics using Spark, preferably PySpark. • • Cloud Technologies/Azure: •Hands-on experience with Azure Data Factory (ADF) for orchestrating data workflows, Azure Databricks for advanced analytics and Azure Synapse Analytics for scalable data warehousing solutions. •Experience with implementing medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake tables in Azure Data Lake Storage (ADLS). •Familiarity with Git and version control for collaborative and efficient project management. •Experience with Microsoft Fabric for seamless data ingestion, processing, and report building within a single unified platform. • Soft Skills & Functional Competencies: •Strong written and verbal communication skills for technical and non-technical audiences. •Demonstrated ability to collaborate with cross-functional teams and take ownership of deliverables. •Critical thinker with a problem-solving mindset and attention to detail. •Passion for AI and continuous learning in the artificial intelligence domain.

Educational Qualifications

•Education: A Bachelor’s or Master’s degree in Engineering. •Certifications: Good to have certifications in Data Engineering (Azure, Databricks, or Fabric).

Work Experience

1-4 years of work experience in data engineering or related fiel