FC

Senior Data Engineer, Databricks & AI Automation

  • Flintex Consulting Pte. Ltd.
  • Singapore
  • On-site
  • Contract
  • SGD 7k–9k / month
  • Posted yesterday
Data & AIBusiness Data AnalysisCollaborate With EngineersPySparkDesign

Senior Data Engineer Databricks and Automation

Data and AI Innovation Team

About the role

Build reliable data pipelines and trusted datasets for analytics, business applications and AI use cases. This hands-on role centres on Databricks, Python, PySpark and SQL. You will also simplify recurring engineering and support activities through automation, using AI where it provides a clear benefit.

Key responsibilities

  • Design, build and maintain production ETL/ELT pipelines using Databricks, PySpark, SQL and Delta Lake.
  • Integrate data from databases, APIs and files, with streaming ingestion where required.
  • Develop curated datasets and data models that meet agreed business definitions, access requirements and consumption needs.
  • Implement data quality checks, reconciliation, incremental processing, schema handling and recovery procedures.
  • Monitor pipeline reliability and data freshness; investigate failures, resolve root causes and improve recovery.
  • Optimise processing performance and job compute usage, partnering with the Platform Engineer on shared settings and controls.
  • Apply version control, automated testing, CI/CD and documentation to support reliable releases and maintainable solutions.
  • Automate repetitive engineering and support tasks; assess AI-assisted approaches against accuracy, security and manual effort saved.

Essential skills and experience

  • Typically 5 or more years of hands-on data engineering experience, including building and supporting production pipelines.
  • Strong Python, SQL and Spark or PySpark skills.
  • Practical Databricks and Delta Lake experience, including job orchestration and troubleshooting. Strong production Spark and lakehouse experience with less Databricks exposure will also be considered.
  • Experience with data modelling, incremental ingestion, data quality, reconciliation and performance tuning.
  • Experience with a cloud data platform and practical use of Git, testing and deployment pipelines.
  • Ability to translate business requirements into maintainable data solutions and collaborate with business and technical teams.

Preferred experience

  • Azure data services and enterprise data integration.
  • Unity Catalog, Auto Loader, Structured Streaming or Databricks deployment tooling.
  • Streaming technologies such as Kafka or Event Hubs.
  • Automation of monitoring, reconciliation or operational support.
  • Practical use of LLMs, AI agents or Databricks AI capabilities to improve engineering workflows.

Role ownership

Own data pipelines, data quality, curated datasets and job-level performance. Partner with the Platform Engineer on shared environments, deployment tooling, access controls and platform monitoring.

What success looks like

Accurate and timely data, reliable pipelines, efficient processing, faster recovery from failures and less manual support effort.

Apply on MyCareersFuture

You apply on MyCareersFuture, where this job was posted. Peter Nowak isn’t the employer and doesn’t check listings.

Get new jobs like this by email

Data & AI · Singapore

Free. Confirm by email first; unsubscribe anytime in one click.

Similar jobs

Senior Data Engineer, Databricks & AI Automation at Flintex Consulting Pte. Ltd. — Singapore | Rivet Commerce job board