Java Lead Software Engineer

Exp: 6 - 9 years

• Design, develop, and maintain efficient and reliable data pipelines using Java, Scala, Apache Spark and Confluent Cloud (Kafka, KStreams, kSQLDB, Schema Registry)
• Leverage Apache Spark (Java/Scala) for large-scale data processing and transformation.
• Experience with building, maintaining and debugging applications and data pipelines using Confluent Cloud (Kafka, KStreams, kSQLDB, Schema Registry).
• Build and optimize data storage solutions using NoSQL databases such as ScyllaDB and/or Cassandra.
• Experienced with AWS services required for Data Engineering such as EMR, ServerlessEMR, AWS Glue, CodeCommit, EC2, S3 etc.
• Familiarity with workflow orchestration tools such as Airflow
• Experience with building and deploying applications using Docker or AWS ECS or AWS EKS
• Well versed with code management using tools like GitHub and CI/CD pipelines and deployment of data pipelines on AWS cloud.
• Implement and manage search and analytics capabilities using AWS OpenSearch and/or Elasticsearch.
• Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver effective solutions.
• Monitor and troubleshoot data pipelines to ensure data quality and performance.
• Implement data governance and data quality best practices.
• Automate data ingestion, processing, and deployment processes.
• Stay up-to-date with the latest data engineering trends and technologies.
• Contribute to the design and architecture of our data platform on AWS.