Senior Data Engineer (Databricks / Spark)

At RSB Automotive Consulting, we are looking for a Senior Data Engineer to support an ongoing data platform optimisation project within a large-scale Databricks Lakehouse environment.

Project overview

The project focuses on improving performance and cost efficiency of existing data pipelines. Following a successful Proof of Concept that evaluated alternative modelling and implementation approaches, the current phase involves rolling out validated optimisations across multiple data domains.

This role offers hands-on involvement in technically meaningful work — from redesigning pipeline architecture to benchmarking performance and documenting measurable improvements.

Your responsibilities

Evaluate and implement data pipeline optimisations, including SCD Type 2 modelling
Refactor existing pipelines to improve performance and cost efficiency
Remove or replace unnecessary components (e.g. DLT) where applicable
Introduce monitoring, tagging and metrics to compare baseline and optimised pipelines
Run parallel executions to validate output consistency and performance
Document architecture decisions, build standards and optimisation rationale

Requirements

Strong hands-on experience with Spark and Databricks
Deep understanding of data modelling, especially trade-offs between SCD Type 1 vs. Type 2
Solid knowledge of Delta Lake and ELT pipeline design
Proven experience in performance tuning and refactoring complex data pipelines
Ability to benchmark solutions and clearly document cost and performance impacts

Additional details

Language: English (C1)
Work mode: 100% remote
Start date: mid-February 2026
Contract duration: 6 months

If you enjoy working on optimisation-heavy data engineering projects and want to see the real impact of your architectural decisions, we would be happy to connect.

📩 Contact RSB Automotive Consulting to learn more about this opportunity.

Senior Data Engineer

About this role