Senior Data Engineer Databricks and Automation
Data and AI Innovation Team
About the role
Build reliable data pipelines and trusted datasets for analytics, business applications and AI use cases. This hands-on role centres on Databricks, Python, PySpark and SQL. You will also simplify recurring engineering and support activities through automation, using AI where it provides a clear benefit.
Key responsibilities
- Design, build and maintain production ETL/ELT pipelines using Databricks, PySpark, SQL and Delta Lake.
- Integrate data from databases, APIs and files, with streaming ingestion where required.
- Develop curated datasets and data models that meet agreed business definitions, access requirements and consumption needs.
- Implement data quality checks, reconciliation, incremental processing, schema handling and recovery procedures.
- Monitor pipeline reliability and data freshness; investigate failures, resolve root causes and improve recovery.
- Optimise processing performance and job compute usage, partnering with the Platform Engineer on shared settings and controls.
- Apply version control, automated testing, CI/CD and documentation to support reliable releases and maintainable solutions.
- Automate repetitive engineering and support tasks; assess AI-assisted approaches against accuracy, security and manual effort saved.
Essential skills and experience
- Typically 5 or more years of hands-on data engineering experience, including building and supporting production pipelines.
- Strong Python, SQL and Spark or PySpark skills.
- Practical Databricks and Delta Lake experience, including job orchestration and troubleshooting. Strong production Spark and lakehouse experience with less Databricks exposure will also be considered.
- Experience with data modelling, incremental ingestion, data quality, reconciliation and performance tuning.
- Experience with a cloud data platform and practical use of Git, testing and deployment pipelines.
- Ability to translate business requirements into maintainable data solutions and collaborate with business and technical teams.
Preferred experience
- Azure data services and enterprise data integration.
- Unity Catalog, Auto Loader, Structured Streaming or Databricks deployment tooling.
- Streaming technologies such as Kafka or Event Hubs.
- Automation of monitoring, reconciliation or operational support.
- Practical use of LLMs, AI agents or Databricks AI capabilities to improve engineering workflows.
Role ownership
Own data pipelines, data quality, curated datasets and job-level performance. Partner with the Platform Engineer on shared environments, deployment tooling, access controls and platform monitoring.
What success looks like
Accurate and timely data, reliable pipelines, efficient processing, faster recovery from failures and less manual support effort.
You apply on MyCareersFuture, where this job was posted. Peter Nowak isn’t the employer and doesn’t check listings.
