Lead Databricks Engineer

Role Overview:

We are looking for a seasoned and visionary Lead Databricks Engineerwith over 14 years of experience to lead the design and implementation of enterprise-grade data architecture, governance frameworks, and scalable data platforms. This role requires deep expertise in modern data technologies, big data ecosystems, and cloud-native solutions. The ideal candidate will be a strategic leader who can bridge business needs with technical solutions, ensuring data integrity, quality, and accessibility across the organization.

Responsibilities:

• Lead the design and implementation of Data Architecture, Data Quality frameworks, and Data Governance initiatives.
• Define data modeling standards and establish foundational best practices across data platforms.
• Champion and promote data quality principles by developing and institutionalizing enterprise-wide standards and methodologies.
• Build and oversee data governance policies, procedures, and frameworks to ensure data integrity, security, and compliance.
• Drive enterprise-wide adoption of self-service data platforms and enablement of organizational data literacy and utilization.
• Define and maintain critical data standards and metadata to facilitate the consistent understanding and use of data as a shared enterprise asset.
• Create reusable templates and establish coding standards for sourcing, collecting, and transforming both streaming and batch data.
• Design scalable data schemas, object models, and flow diagrams to effectively structure, process, and integrate data across systems.
• Provide architectural strategy, assessments, and roadmaps to guide enterprise data management practices.
• Offer hands-on expertise in architecting and managing Big Data platforms and Data Lake technologies (e.g., AWS S3, Hive), along with exposure to ML and Data Science tools.
• Implement and manage industry-standard tools and frameworks such as Databricks, Delta Lake, Apache Spark, Airflow, Hive Metastore, Redshift, Kafka, Kubernetes, Docker, and CI/CD pipelines.
• Translate complex big data and analytics requirements into high-performance, scalable data models and guide analytics engineers on implementing these models.
• Define reusable templates and processes for data modeling, data flow design, and system integration.
• Lead, mentor, and upskill data analytics teams by promoting best practices, modern technologies, and effective processes in data platform engineering.

Tech Skills:

• 6+ years of hands-on experience in Data Warehouse, ETL, Data Modeling & Reporting
• 5+ years of hands-on experience in productionizing and deploying Big Data platforms and applications.
• Minimum of 3 years of hands-on DataBricks experience required.
• Hands-on experience working with: Relational/SQL, distributed columnar data stores/NoSQL databases, time-series databases, Spark streaming, Kafka, Hive, Delta Parquet, Avro, and more
• Extensive experience in understanding a variety of complex business use cases and modeling the data in the data warehouse
• Highly skilled in SQL, Python, Spark, AWS S3, Hive Data Catalog, Parquet, Redshift, Airflow, and Tableau or similar tools
• Proven experience in building a Custom Enterprise Data Warehouse or implementing tools like Data Catalogs, Spark, Tableau, Kubernetes, and Docker
• Demonstrated industry leadership in the fields of Data Warehousing, Data Science, and Big Data related technologies
• Design and Implement Machine learning models with Tensorflow ,Keras and pytorch And MLops