Data Engineer

Job Description

Role Overview and Expected Deliverables

The engineer will design, build and operate scalable, secure and analytics‑ready data pipelines and models across GCP BigQuery and AWS, delivering:
- Airflow‑orchestrated data workflows and automations.
- DBT Cloud transformations, testing frameworks and environment governance.
- Robust CI/CD and Infrastructure as Code (IaC) deployments (GitHub Actions, Bitbucket, Bamboo, Terraform or CloudFormation).
- Performance optimisation across data pipelines, SQL, BigQuery and AWS data services.
- High‑quality technical documentation, production support, incident response and handover to internal teams.
- Close collaboration with cross‑functional stakeholders including analytics, product and platform engineering.

Qualifications

Must have skills

GCP, BigQuery, Cloud Storage, Apache Airflow, dbt, Python, SQL, data pipelines, ingestion, transformation, Git/Bitbucket, data quality checks

Good to have skills

Dataplex, Collibra integration, CI/CD, Grafana, Splunk, Power BI data preparation, financial services data

Core Technical Requirements

Cloud & Platform Engineering:
- Strong proficiency across AWS and GCP.
- On AWS: IAM roles and policies, Lambda, S3, RDS, Redshift.
- On GCP: BigQuery, workload optimisation, cost management.

Data Engineering Tooling:
- Hands‑on experience building data pipelines for data warehouse environments.
- Strong SQL capability including analytical and window functions.
- Advanced dbt Cloud experience (models, SCDs, tests, macros, release management).
- Airflow workflow design including idempotency, backfills and monitoring.

DevOps, CI/CD & IaC:
- Experience with CI/CD (GitHub, Bitbucket, GitHub Actions, Bamboo).
- Infrastructure as Code using Terraform or CloudFormation.
- Familiarity with data quality frameworks and automated testing of pipelines.

Programming & Modelling:
- Proficiency in Python.
- Strong data modelling expertise: facts/dimensions, SCD patterns, performance optimisation.

Security, Data Governance & Operational Resilience:
- Solid grounding in cloud security best practices and least‑privilege IAM.
- Understanding of data access controls, regulated environments and governance expectations.

Delivery Approach & Ways of Working- Experience working within Agile delivery frameworks.
- Use of JIRA and Confluence for delivery management and documentation.
- Proven track record collaborating effectively in multi‑disciplinary teams across engineering, product and analytics.

Specialisations - BigQuery performance tuning and cost optimisation.
- DBT Cloud governance (contracts, tests, macros, release processes).
- Airflow workflow optimisation and monitoring.
- Cross‑cloud pipelines and data movement (AWS ↔ GCP).
- Security, IAM and data‑access design for regulated financial services environments.

Experience & Other- 8 or more years experience in data engineering or equivalent technical roles.
- Strong problem‑solving capability, comfortable troubleshooting pipeline failures and performance issues.
- Strong communication and stakeholder management skills.
- Self‑starter able to lead components independently while partnering closely with internal teams.
- Experience supporting production environments, including incident response and root‑cause analysis.
- All intellectual property rights retained by client

Additional Information

Open to hire for Gurgaon and Hyderabad location as well

At Endava, we’re committed to creating an open, inclusive, and respectful environment where everyone feels safe, valued, and empowered to be their best. We welcome applications from people of all backgrounds, experiences, and perspectives—because we know that inclusive teams help us deliver smarter, more innovative solutions for our customers. Hiring decisions are based on merit, skills, qualifications, and potential. If you need adjustments or support during the recruitment process, please let us know.