Key Responsibilities
•
Engage with business stakeholders to understand requirements and translate them into data models, datasets and pipelines.
•
Design and develop data models and transformation pipelines to integrate, clean, harmonize and curate data from multiple sources.
•
Build and maintain scalable and reliable data pipelines using Databricks, Apache Spark and SQL.
•
Design curated datasets optimized for reporting, analytics and data science use cases.
•
Implement data quality checks, validation and reconciliation to ensure data accuracy and consistency.
•
Monitor and troubleshoot data pipeline performance and operational issues.
•
Document data models, datasets, transformations and pipeline dependencies.
•
Collaborate with data engineers, analysts and data scientists to deliver solutions aligned with business requirements.
Must-Have Requirements
•
Strong proficiency in SQL, relational databases and data warehousing.
•
Strong data modelling skills, including dimensional modelling and analytical dataset design.
•
Proficiency in Python for data processing and pipeline development.
•
Hands-on experience with Databricks, Apache Spark and Delta Lake.
•
Experience building and maintaining data ingestion and transformation pipelines.
•
Ability to engage business stakeholders and translate requirements into technical data solutions.
Good to Have
•
Experience developing ETL/ELT pipelines and data transformation workflows.
•
Experience with AWS data services and cloud-based data platforms.
•
Familiarity with Git, CI/CD and data pipeline orchestration.
•
Experience with Databricks Workflows, Unity Catalog and Delta Lake.
•
Experience designing data products for BI and self-service analytics.
•
Knowledge of data governance, access control and data quality practices.
•
Experience in government or other highly regulated environments.
•
Degree in Computer Science, Information Technology, Data Engineering or a related discipline.
You apply on MyCareersFuture, where this job was posted. Peter Nowak isn’t the employer and doesn’t check listings.
