Data Engineer

About the Role

The data engineer role is changing. The traditional pattern - build a pipeline, hand it off, wait for someone else to figure out if the data is right - doesn't work anymore. The engineers who create the most value now are the ones who go deep into the business domain, understand the problem firsthand, and use AI tools to move from question to working solution in hours instead of sprints.

The Analytics team is looking for a Senior Data Engineer who operates as a hybrid technical product owner: someone who builds and maintains the data platform on Azure Databricks, but who also sits with business stakeholders, interrogates the problem, and owns the outcome end-to-end. This role bridges data engineering (70%) and ML/AI (30%), not a pure data engineer, but someone who bridges both. You won't wait for requirements to be handed to you. You'll go find them, validate them, and ship the solution using Claude Code and agentic development patterns to collapse the distance between understanding a business problem and solving it in production.

Each DE owns one workstream (marketing, ops, physician growth, etc.) alongside a technical product owner. This is not a solo role, not a lead role. You work WITH a technical product owner, not as the owner themselves.

This is what it means to shift the engineer left into the business domain. You're not a pipeline builder waiting for a ticket. You're the person who understands how physician network growth, member engagement, and operational performance actually work and who builds the data infrastructure that makes the entire Analytics team faster, sharper, and more impactful.

What You'll Do

Own the Data Platform

• Design, build, and operate the data platform on Azure Databricks: ingestion, transformation, storage, and serving layers that power analytics, AI models, and operational reporting.
• Build and maintain data pipelines across the ecosystem: Salesforce, SQL Server, Snowflake, third-party sources, and the new cloud-native payments platform.
• Engineer for quality and trust with validation checks, anomaly detection, lineage tracking, and documentation that ensure every downstream consumer can rely on the data.
• Write clean, version-controlled, production-grade code. Think like a software engineer building a product, not a script runner maintaining jobs.

Go Deep Into the Business Domain

• Partner directly with business stakeholders across physician growth, member services, finance, and operations to understand how data drives decisions and then build for those decisions, not for abstract requirements.
• Act as a technical product owner for your domain areas: own the backlog, prioritize based on business impact, and ship iteratively without waiting for a PM to sequence your work.
• Translate ambiguous business questions into data models, feature tables, and curated datasets that analysts and data scientists can build on immediately.
• Close the loop: follow your data through to the dashboard, the model, or the operational workflow and validate that it's actually driving the outcome.

Drive AI-First Engineering Practices

• Use Claude Code and agentic development as your primary workflow: AI-driven pipeline generation, automated testing, rapid prototyping to ship at a pace that would be impossible with traditional approaches.
• Build data infrastructure that is AI-ready: well-documented, semantically clear, and structured so that AI tools and agents can reason over it effectively.
• Scout, evaluate, and adopt emerging AI tools and platforms that make the data team faster by separating real value from hype with hands-on testing.
• Share what you learn. Document patterns, run demos, and help the broader team adopt AI-first workflows with confidence.

Real Use Case - Doctor Recruitment Pipeline

Your work may involve:

• Sales team tracks every doctor engagement (email, phone, cold calls, martech tools)
• Pull engagement data from multiple source systems into Databricks via APIs
• Model and layer the data, serve to Power BI dashboards
• Expose dataset to Claude so analytics team can ask questions directly
• Build engagement scoring model: Claude generates logic (vibe coding), Python validates the scoring, scores written back into Databricks

Who You Are

• A data engineer who refuses to stay in the technical silo. You go find the business problem, not wait for it to arrive as a Jira ticket.
• Someone who thinks like a product owner: you prioritize by impact, ship incrementally, and own the outcome, not just the pipeline.
• The kind of engineer who's already using AI tools to write, test, and deploy code before anyone asked you to and who knows when the output needs a human eye.
• Equally comfortable writing a Spark transformation, debugging a Salesforce data sync, presenting findings to leadership, and pushing back on a vague requirement until it's sharp enough to build against.
• Pragmatic over perfectionist. You optimize for business impact and speed to value, not theoretical elegance.
• A strong collaborator who elevates the people around them through clear communication, reusable patterns, and generous knowledge sharing.

Required Qualifications

• BS in Computer Science, Data Science, or related field; 6+ years in data engineering or a hybrid data engineering/analytics role.
• Deep hands-on experience with Azure Databricks: notebooks, Delta Lake, Unity Catalog, and production-scale pipelines. Databricks 3+ years is non-negotiable.
• Strong Python and SQL; experience with PySpark and distributed data processing. Python is essential for validating AI-generated logic.
• Built and operated data pipelines that serve analytics, ML models, and operational systems, not just batch ETL jobs.
• Worked directly with business stakeholders to define requirements, shape data products, and deliver measurable outcomes.
• Active, daily use of AI coding tools (Claude Code, Copilot, or similar) as a force multiplier. Claude and AI tooling in active daily use (team already works this way).
• Strong communication skills with a track record of presenting technical work to non-technical audiences.
• Power BI familiarity for serving downstream BI team.
• Salesforce/CRM data integration experience, directly relevant to use case.

Preferred Qualifications

• Experience with engagement scoring or machine learning model pipelines
• Familiarity with agentic development patterns and AI-assisted coding
• Background in healthcare or mission-driven industries
• US-based preferred, not a hard requirement

Team Context

• Team specifics: DEs, Power BI devs, data scientists, analytics, technical product owner
• Adding headcount at fiscal year start (July)
• Each DE owns one workstream (marketing, ops, etc.) alongside a technical product owner
• Not a solo role, not a lead role
• Databricks in place 3+ years, stable Azure environment (not greenfield)
• Work is delivered in sprints — each DE ships iteratively within their workstream alongside a technical product owner.