Area Manager-Data Engineering-Cloud Data Engineering

Design, develop, and maintain scalable and efficient data pipelines using PySpark and Databricks

Perform data extraction, transformation, and loading (ETL) from diverse structured and unstructured data sources

Develop and maintain data models, data warehouses, and data marts in Databricks

Proficient in Python, Apache Spark, and PySpark 

Integrate third party data from multiple sources with internal data 

Write and optimize complex SQL queries for high performance and scalability across large datasets

Collaborate closely with data scientists, analysts, and business stakeholders to gather and understand data requirements.

Ensure data quality, consistency, and integrity throughout the data lifecycle using validation and monitoring techniques.

Develop and maintain modular, reusable, and well-documented code and technical documentation for data workflows and processes.

Implement data governance, security, and compliance best practices.

Strong hands-on experience with PySpark, Databricks, and distributed data processing.

Proficiency in Python, with deep knowledge of Pandas and NumPy for data manipulation and analysis.

Strong foundation in Data Structures and Algorithms for writing efficient and optimized code.

Solid understanding of ETL processes, data modeling, and data warehousing concepts.

Experience with SQL and performance tuning for large-scale datasets.

Familiarity with data visualization tools such as Power BI, or Tableau.

Knowledge of version control systems like Git and collaborative development workflows.

Strong problem-solving skills, attention to detail, and ability to work independently or in a team.

5+ years of relevant experience in Data Engineering tools

Programming Languages: Python and SQL

Data Processing Tools: pandas, NumPy, PySpark

Cloud Platforms: Databricks (for scalable computing resources)

Version Control & Collaboration: Git, GitHub, GitLab

Deployment and Monitoring: Databricks, Github Actions