DataBricks - Associate Director

Mandatory Skills:

• Bachelor’s or higher degree in Computer Science, Information Technology, or a related discipline; or equivalent experience with minimum 12+ years overall work experience.
• Extensive hands-on experience with Databricks and Apache Spark, including PySpark, SQL, and Scala for data processing and analytics.
• Proven expertise in designing and implementing efficient data ingestion pipelines using Databricks, managing large-scale datasets from various sources.
• Experience delivering proofs of concept to demonstrate the capabilities of Databricks to stakeholders.
• Skilled in developing scalable and reusable frameworks for data ingestion and transformation using Databricks features such as Delta Lake.
• Strong understanding of cloud architectures, particularly within Azure and Databricks integration, including Azure Databricks, Azure Data Lake Storage, and Azure Blob storage.
• Experience in implementing CI/CD pipelines and DevSecOps practices in Databricks environments.
• Proficiency in cloud migration methodologies and processes pertinent to data engineering.
• Strong communication skills, both written and verbal, with the ability to present complex technical information in a clear and concise manner.
• Self-motivated, proactive working style with a strong sense of ownership and problem-solving capability.
• Collaborative team player with experience in mentoring and guiding other engineers on best practices in data engineering.
• Experience in using Big Data file formats (e.g., Parquet, Avro) and compression techniques.
• Familiarity with data governance and metadata management tools within Databricks and Azure environments.

• Design, build, and maintain the data infrastructure utilizing Databricks and its integration within Azure ecosystems.
• Develop and optimize data pipelines to extract, transform, and load data from diverse sources into Databricks-supported data lakes.
• Implement and manage data security and privacy measures, ensuring compliance with industry standards and regulations.
• Monitor and troubleshoot data workflows and optimize data storage and processing for performance and cost efficiency using Databricks.
• Collaborate with cross-functional teams to ensure data quality and availability for business stakeholders and analytics teams.
• Provide strategic technical guidance and support for Databricks initiatives, ensuring alignment with business goals.
• Innovate and implement solutions using Databricks and Spark technologies for processing and analyzing big data.
• Participate in projects focused on cloud architecture improvements, including planning and executing migrations to Databricks and Azure.