DataBricks / AWS Redshift

Virtusa.com

Office

Full Time

DataBricks / AWS Redshift - (CREQ234999)

Description

Key responsibilities include designing developing and maintaining scalable data pipelines using Apache Spark on Databricks Writing efficient and production ready PySpark or Scala code for data transformation and ETL processes Integrating data from various structured and unstructured sources into a unified platform Implementing Delta Lake and managing data versioning updates and schema evolution Optimizing data processing workflows for performance scalability and cost efficiency Collaborating with data scientists analysts and business stakeholders to deliver high quality datasets Implementing data quality checks validation routines and logging mechanisms Monitoring and debugging production jobs using Databricks jobs notebooks and clusters Ensuring security privacy and compliance standards are met throughout the data lifecycle Providing guidance and mentorship to junior team members Required skills and qualifications include four to six years of experience in Big Data development Hands on experience with Databricks including Workflows Notebooks Delta Live Tables and Unity Catalog Strong programming skills in PySpark and or Scala Solid understanding of Delta Lake architecture Proficiency in SQL for data analysis and transformation Experience with cloud platforms such as Azure including Azure Data Lake Data Factory and Synapse or AWS including S3 Glue and Redshift Familiarity with CI and CD for Databricks deployments such as using GitHub Actions or Azure DevOps Knowledge of data governance cataloguing and security best practices Experience working in an Agile or Scrum environment Required skills include Databricks Redshift and SQL Preferred skills include Python and Snowflake Additional preferred experience includes working with Databricks Unity Catalog and Delta Live Tables exposure to machine learning workflows in Databricks and experience with orchestration or messaging tools such as Apache Airflow or Kafka Certifications such as Databricks Certified Data Engineer Associate or Professional Azure or AWS certification are also desirable

Primary Location

: IN-TN-Chennai

Schedule

: Full Time

Employee Status

: Individual Contributor

Job Type

: Experienced

Travel

: No

Job Posting

: 09/10/2025, 12:06:44 PM