About the Role
We are seeking an experienced Senior Data Engineer to design, develop and maintain large-scale data processing solutions for aviation systems. You will build reliable batch and real-time data pipelines, integrate multiple data sources and implement scalable data storage, processing and recovery mechanisms.
Key Responsibilities
•
Design, develop and maintain batch and real-time data pipelines using Apache Spark, Apache Flink and Apache Kafka.
•
Implement distributed data processing solutions using Java or Scala and SQL.
•
Integrate multiple data sources and S3-compatible object storage for data ingestion and processing.
•
Manage data schemas, metadata, lineage and data quality across processing pipelines.
•
Deploy, maintain and scale data processing workloads in Kubernetes environments.
•
Implement automated testing, GitLab CI/CD pipelines, monitoring and operational troubleshooting.
•
Establish checkpointing, recovery, data retention and lifecycle management mechanisms.
•
Support high availability, backup, disaster recovery and data synchronisation across multiple sites.
Requirements and Qualifications
•
5–8 years of relevant experience in data engineering, distributed systems or large-scale data platform development.
•
Strong hands-on experience designing and developing batch and streaming data pipelines using Apache Spark, Apache Flink and Apache Kafka.
•
Proficiency in Java or Scala for distributed data processing, together with strong SQL skills.
•
Experience deploying and scaling data workloads using Docker and Kubernetes.
•
Familiarity with S3-compatible object storage, data schemas, metadata management, data lineage and data quality controls.
•
Experience with checkpointing, fault recovery, data retention and disaster recovery mechanisms.
•
Hands-on experience with GitLab, CI/CD pipelines, automated testing and monitoring.
•
Strong analytical, troubleshooting and problem-solving skills.
Only shortlisted candidates will be notified.
(EA Reg No: 20C0312)
You apply on MyCareersFuture, where this job was posted. Peter Nowak isn’t the employer and doesn’t check listings.
