Data Engineer Manager

We are seeking an experienced Data Engineering Manager to lead a high-performing team that designs, builds, and operates our enterprise data platform. This role blends hands-on technical leadership with people management, Agile delivery, and data platform architecture. You will guide the development of scalable ingestion pipelines, enforce data quality and governance, and partner with product and business stakeholders to deliver reliable data products that enable analytics and data science across the organization.

Position Details

Technical Responsibilities

• Design, build, and maintain scalable data pipelines and ETL/ELT workflows using Python and SQL.
• Architect and implement data ingestion patterns for batch, streaming, and hybrid workloads.
• Develop and optimize solutions on Databricks, leveraging Spark, Delta Lake, Unity Catalog, DLT, and Workflows.
• Define and implement medallion (Bronze/Silver/Gold) architecture patterns for transformation and governance.
• Establish and enforce data quality standards, monitoring, alerting, and observability across the data platform.
• Collaborate with data scientists, analysts, and business stakeholders to understand requirements and deliver reliable data products.
• Contribute to and maintain data platform architectural diagrams and extensibility guidance.

Leadership Responsibilities

• Identify and partner with a Product Owner to establish, curate, and execute a transformation backlog.
• Co-organize and co-host Sprint ceremonies; lead team solutioning sessions and provide estimates.
• Lead technical decision-making and prepare demos/walk-throughs for technical and non-technical audiences.
• Coordinate with DevOps and adjacent teams on code merges, testing, and release management.
• Document, organize, and lead deployments from development through production; ensure best practices and guidelines are adhered to.
• Maintain and report scrum metrics and feature forecasts daily; proactively escalate risks and recommendations to the VP of Engineering as appropriate.
• Interview and onboard new hires; approve and document time off; conduct annual performance reviews and salary adjustment recommendations.
• Hold regular 1:1s to discover and resolve personnel issues; mentor and coach team members on career growth.
• Create, own, and educate others on the client sandbox solution architecture; establish and maintain a backlog of changes (application growth, data refreshes, and planned curations).
• Design data staging frameworks based on real-world scenarios (e.g., month-end reconciliation, homeowner settlement/warranty deed, posting to GL, go-dark practice run—architecture only).
• Ensure Databricks and client read-only connectivity; develop code that can model extensibility development.

Qualifications

Required Qualifications

Technical Skills

• Big Data Platforms: Hands-on experience with enterprise data platforms; strong preference for Databricks (Unity Catalog, Delta Lake, DLT, Workflows, Spark optimization).
• Python: Strong to expert-level proficiency including PySpark, pandas, data validation libraries (e.g., Pydantic, Great Expectations), and production-grade coding practices.
• Data Ingestion: Expert-level experience with Auto Loader, CDC, streaming ingestion, API integrations, and file-based batch processing.
• SQL: Advanced skills (complex joins, window functions, CTEs, query optimization) across analytical and OLTP workloads (SparkSQL, T-SQL or similar).
• Platform Architecture: Understanding of lakehouse architecture, data modeling principles (Kimball, Data Vault), and data governance frameworks.

Experience

• 5+ years in data engineering or related roles, with 2+ years in a formal people-management capacity.
• Proven track record delivering production-grade data pipelines at scale and leading cross-functional initiatives.
• Experience with CI/CD for data pipelines and infrastructure-as-code practices.

Soft Skills

• Strong, proactive daily communication and ownership mindset.
• Ability to demonstrate technical functionality to technical and non-technical audiences.
• Clear risk identification, mitigation planning, and follow-through.
• People leadership with empathy, accountability, and coaching orientation.

Preferred Qualifications

• Databricks certifications (Data Engineer Associate/Professional).
• Experience with major cloud platforms (AWS or Azure) and native data services.
• Familiarity with orchestration tools (Airflow, Prefect, Databricks Workflows).
• Experience driving data migration or multi-source integration programs.
• Knowledge of real-time streaming technologies (Kafka, Event Hubs, Kinesis).
• 2+ years managing engineering teams; formal training or certifications in Agile/Scrum (e.g., CSM, SAFe).