Databricks Lead Data Engineer

Job Overview:

As a Databricks Lead Data Engineer, you are expected to lead the development team and have strong development and programming skills in Python, SQL, Pyspark, Databricks, Notebook Repositories, Medallion Architecture, CI/CD, and ETL platforms associated with Data Lakehouse solutions. You are expected to be experienced and fluent in agile development and agile tools.

You will work with architects and infrastructure teams to develop, test, deploy and troubleshoot your code as well as provide input into solutions and design of the system. You will collaborate with some of the best talent in the industry to create and implement innovative high-quality solutions focused on our clients' business needs.

Perficient is always looking for the best and brightest talent and we need you! We’re a quickly-growing, global digital consulting leader, and we’re transforming the world’s largest enterprises and biggest brands. You’ll work with the latest technologies, expand your skills, and become a part of our global community of talented, diverse, and knowledgeable colleagues.

• Lead the technical planning and requirements gathering.
• Serve as a technical lead and mentor. Provide technical support and leadership in the development and continual improvement of service.
• Develop and maintain effective working relationships with team members.
• Demonstrate the ability to adapt and work with team members of various experience levels.
• Work with the data engineering team to define and develop data ingestion, validation, transformation, and data engineering code.
• Lead the design, development, testing and deployment of scalable batch/streaming data pipelines using Databricks. Data jobs include data ingestion, cleansing, transformation, aggregation, governance, and sharing.
• Define and enforce best practices for Databricks usage, including notebook development, job orchestration, metadata-driven frameworks, data quality checks, and optimization of spark job performance.
• Develop Databricks platform components using Python, SQL, Pyspark, Notebook Repositories (medallion architecture L2, L3, L4), and CI/CD.
• Integrate Databricks solutions with cloud platforms like Azure (ADF, ADLS, Event Hub, Azure DevOps) or AWS (S3, Lamda, Glue).
• Design and implement analytical data models and star schemas for BI and AI usage.
• Collaborate with data scientists and analysts to meet data requirements.
• Implement logging, monitoring and exception handling for data workflows.
• Participate in code reviews and CI/CD pipeline development.
• Document code artifacts and participate in developing user documentation and run books
• Troubleshoot deployment to various environments and provide test support.
• Participate in design sessions, demos, and prototype sessions, testing and training workshops with business users and IT associates.
• Minimum 5 years of hands-on Databricks experience using Python, SQL, Pyspark, Notebook Repositories (medallion architecture L2, L3, L4), CI/CD.
• Passionate developer with 8+ years of data engineering experience, with at least 3 years in a lead or senior data engineer role.
• Minimum 5 years of Databricks experience in developing large scale data processing/data storage/data distribution systems.
• Experience with AWS and/or Azure cloud environment • Experience with various Notebook Repositories (medallion architecture L2, L3, L4) as well as unit testing frameworks.
• DevOps experience with Databricks CI/CD (Databricks Asset Bundles) and Git
• Minimum 5 years of working with large multi-vendor environment with multiple teams as a part of a project
• Minimum 5 years of working with a complex Big Data/Lakehouse environment
• Minimum of 5 years of experience with JIRA/GitHub/Git and other code management toolsets
• Experience with ELT/ETL development, patterns and tooling is recommended
• Experience with SQL including Postgres, RDBMS platforms
• Experience with various IDE and code repositories as well as unit testing frameworks.
• Flexible and adaptable attitude, disciplined to manage multiple responsibilities and adjust to varied environments.
• Ability to produce high quality products within deadlines
• Strong written and oral communication skills; Ability to communicate effectively with technical and non-technical staff.
• Excellent problem-solving skills.
• Be independent and self-driven.
• Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field.
• Work is expected to be out of the local Perficient office.
• Must be open to travel and work at client site when required.

Preferred Skills & Education

• Certification in Databricks, Spark, AWS, Azure or other cloud platform
• Experience with ETL/ELT data platforms such as Informatica, Data Stage, Fivetran, Matillion, dbt, etc.
• Familiarity with ML capabilities such as MLflow, TensorFlow, or Scikit-learn is a plus
• Master’s degree in Computer Science or related field
• Client-facing or consulting experience is highly preferred.

Perficient full-time employees receive complete and competitive benefits. We offer a collaborative work environment, competitive compensation, generous work/life opportunities and an outstanding benefits package that includes paid time off plus holidays. In addition, all colleagues are eligible for a number of rewards and recognition programs including billable bonus opportunities. Encouraging a healthy work/life balance and providing our colleagues great benefits are just part of what makes Perficient a great place to work. 

Who We Are

Perficient is a leading global digital consultancy. We imagine, create, engineer, and run digital transformation solutions that help our clients exceed customers’ expectations, outpace competition, and grow their business. With unparalleled strategy, creative, and technology capabilities, our colleagues bring big thinking and innovative ideas, along with a practical approach to help our clients – the world’s largest enterprises and biggest brands succeed.

What We Believe

At Perficient, we promise to challenge, champion, and celebrate our people. You will experience a unique and collaborative culture that values every voice. Join our team, and you’ll become part of something truly special.

We believe in developing a workforce that is as diverse and inclusive as the clients we work with. We’re committed to actively listening, learning, and acting to further advance our organization, our communities, and our future leaders… and we’re not done yet.

Perficient, Inc. proudly provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, gender, sexual orientation, national origin, age, disability, genetic information, marital status, amnesty, or status as a protected veteran in accordance with applicable federal, state and local laws. Perficient, Inc. complies with applicable state and local laws governing non-discrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including, but not limited to, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training. Perficient, Inc. expressly prohibits any form of unlawful employee harassment based on race, color, religion, gender, sexual orientation, national origin, age, genetic information, disability, or covered veterans. Improper interference with the ability of Perficient, Inc. employees to perform their expected job duties is absolutely not tolerated.

Disability Accommodations:

Perficient is committed to providing a barrier-free employment process with reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or accommodation due to a disability, please contact us.

Disclaimer:  The above statements are not intended to be a complete statement of job content, rather to act as a guide to the essential functions performed by the employee assigned to this classification.  Management retains the discretion to add or change the duties of the position at any time. 

Perficient is the global AI-first consultancy. Our team of strategists, designers, technologists, and engineers partners with the world’s most innovative enterprises and admired brands to deliver real business results through the transformative power of AI. As part of our AI-First strategy, we empower every employee to build AI fluency and actively engage with AI tools to drive innovation and efficiency. We break boundaries, obsess over outcomes, and shape the future for our clients. Join a company where bold ideas and brilliant minds converge to redefine what’s possible - while building a career filled with growth, balance, and purpose.