Data bricks Lead Data Engineer - R01565550
Posted about 2 months ago
Job Description – Data Engineer (Databricks)
Company: Brillio Technologies
Role: Data Engineer – Databricks
Experience: 5–10 Years
Location: India (Hybrid/3 days)
Employment Type: Full-time
About Brillio
Brillio is a leading digital transformation partner delivering innovative solutions across Data, Cloud, AI, and Digital Engineering. We help global enterprises unlock value through cutting-edge technology and data-driven insights.
Role Overview
We are looking for a hands-on Data Engineer with strong Databricks expertise to design, build, and scale modern data platforms. The ideal candidate will have experience in PySpark, Delta Lake, and cloud-native data engineering and will work closely with analytics, product, and business teams to deliver high-performance data pipelines.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark
- Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers)
- Develop data ingestion frameworks from multiple sources (batch & real-time)
- Implement Delta Lake features including schema evolution, ACID transactions, and optimization
- Create and manage Databricks notebooks, workflows, and jobs for production pipelines
- Ensure data quality, governance, and lineage across pipelines
- Optimize Spark jobs for performance (partitioning, caching, tuning)
- Collaborate with cross-functional teams (Data Scientists, Analysts, Architects)
- Support production deployments, monitoring, and troubleshooting of data pipelines
- Contribute to migration of legacy data platforms to modern cloud-based architecture
- Strong experience in:
- Databricks (Workflows, Notebooks, Jobs)
- PySpark / Spark SQL
- SQL & Data Warehousing concepts
- Hands-on experience with:
- Delta Lake / Lakehouse architecture
- Data pipeline development & orchestration
- Good understanding of:
- Data modeling, ETL/ELT processes, SCD concepts
- Experience in at least one:
- AWS (S3, Glue, EMR) / Azure (ADF, ADLS, Synapse)
- Workflow tools:
- Airflow / Databricks Workflows
- Version control:
- Git / CI-CD pipelines
- Experience with:
- Streaming frameworks (Kafka / Spark Streaming)
- DBT / Snowflake / Redshift / Fabric
- Exposure to:
- Unity Catalog, Autoloader, Delta Live Tables
- 5–10 years of experience in Data Engineering
- Minimum 2–4 years of hands-on Databricks experience
- Proven experience building enterprise-scale data platforms and pipelines
- Strong problem-solving and analytical skills
- Ability to work in a fast-paced, agile environment
- Good communication and stakeholder management
- Work on cutting-edge data & AI programs
- Opportunity to build large-scale data platforms
- Collaborative and innovation-driven culture
Technical Skills (Must-Have)
Cloud & Tools
Good to Have
Experience Required
Soft Skills
Why Join Brillio
Other open roles at Brillio(6)
From data ingestion and transformation to advanced analytics and visualization, we provide end-to-end solutions to help you drive business growth.
Key team members
Joseph Howard
John Johnson
Phil Jennings
Ashok Venkatraman
Jobr aggregates jobs directly from company career portals — no middlemen. Our team applies on your behalf with AI-tailored resumes, reviewed by a human before submission.