Key Responsibilities
•
Design, develop and maintain scalable ETL/ELT data pipelines using Databricks, Python, PySpark, SQL and Delta Lake.
•
Integrate data from multiple sources, including databases, APIs and files, with streaming ingestion where required.
•
Develop and maintain curated datasets and data models to support business reporting, analytics and AI applications.
•
Implement data quality checks, data validation, reconciliation and incremental data processing.
•
Monitor data pipelines, investigate failures, troubleshoot issues and ensure data accuracy and availability.
•
Optimise data processing performance, pipeline execution and compute utilisation.
•
Implement version control, automated testing, CI/CD and technical documentation.
•
Identify and automate repetitive data engineering and support activities.
•
Explore practical AI
You apply on MyCareersFuture, where this job was posted. Peter Nowak isn’t the employer and doesn’t check listings.
