Data Flow Engineer

We are looking for an external consultant, Data Flow Engineer, to join our team and support data integration, big data streaming, and flow management initiatives. The role focuses on designing, implementing, and maintaining complex data flows using modern data engineering tools and platforms, with emphasis on Apache NiFi, data streaming, and real-time data processing within enterprise environments.

Key Responsibilities

• Design, implement, test, and maintain complex data flows using Apache NiFi (Cloudera DataFlow)
• Build and optimise CDC-based data pipelines (real-time / near real-time)
• Integrate data flows with external systems using REST APIs, JDBC, Kafka, and other protocols
• Manage data ingestion, transformation, enrichment, routing, and delivery processes
• Configure and maintain data schemas, metadata, and lineage
• Implement and manage security and governance policies for data flows
• Monitor, troubleshoot, and optimise performance and reliability of data pipelines
• Collaborate with data engineers, architects, and business stakeholders
• Create and maintain documentation, SOPs, and runbooks
• Participate in platform upgrades, migrations, and continuous improvement activities

Requirements

• Minimum 8 years of overall IT experience
• Minimum 6 years of experience in similar positions
• Experience working with Apache NiFi / Cloudera DataFlow environments
• Strong experience in data integration and data flow design
• Advanced Python programming skills (data processing, automation, integrations)
• Strong experience building integrations using REST APIs (OAuth/JWT, error handling, rate limiting)
• Hands-on experience with CDC pipelines and relational database integrations
• Experience with Apache Kafka and data streaming architectures
• Knowledge of Apache Avro and schema evolution concepts
• Experience with Apache Iceberg (tables, schema evolution, partitioning)
• Knowledge of data governance tools such as Apache Atlas and Apache Ranger
• Experience working in CDP (Cloudera Data Platform) environments
• Strong analytical and problem-solving skills
• Fluency in English (minimum B2 level)
• Minimum education level equivalent to Level 6 qualification
• Valid Security Clearance is mandatory

Certifications (mandatory):

• At least one of the following:
• Cloudera Certified Developer for Apache NiFi (or equivalent)
• Cloudera DataFlow (CFM) certification (or equivalent internationally recognised certification)

You will be a great candidate for us if you

• Have strong hands-on experience with Apache NiFi in production environments
• Are comfortable working with complex data integration and streaming architectures
• Can design and optimise real-time data pipelines
• Have a structured and detail-oriented approach to data engineering
• Are able to work independently and deliver high-quality solutions
• Collaborate effectively with technical and business teams

Benefits

Why join inventYOU

• Work on complex and large-scale data engineering environments
• Contribute to modern data integration and streaming architectures
• Collaborate with experienced data engineers and architects
• Be part of technically challenging and high-impact projects
• Develop your expertise in big data, streaming, and data flow technologies