ES

Lead HPC Systems Engineer

  • EPAM Systems
  • New York, United States
  • On-site
  • Full-Time
  • $65,000 - 102,000 per year
  • Posted 6 days ago
Other software

Salary: $65,000 - 102,000 per year

Requirements:

  • Experience operating production HPC, GPU/accelerator clusters, or large-scale distributed compute in a reliability-focused engineering role
  • Strong Linux and networking skills, including debugging processes, containers, storage I/O, DNS, connectivity, and authentication
  • Ability to correlate telemetry across multiple cloud environments using tools such as Prometheus, Grafana, OpenTelemetry, or centralized logging
  • Proficiency maintaining automation in a language such as Python or Go, with experience owning production investigations through root cause analysis and lasting, verified fixes
  • Helpful experience with HPC scheduling or cluster orchestration such as Slurm or Kubernetes, and understanding how infrastructure health affects workload placement and completion
  • Helpful experience supporting AI training or accelerator-intensive workloads, including GPU/TPU health, distributed-job failures, interconnect issues, data-loading bottlenecks, or checkpointing
  • Familiarity with cloud providers such as AWS, Google Cloud Platform, CoreWeave, or OCI; experience with every listed provider is not required
  • Helpful knowledge of HPC storage and networking, including NFS/POSIX, Lustre, object storage, RDMA/InfiniBand or RoCE, as well as infrastructure-as-code and deployment tools such as Terraform, Helm, or cluster operators
  • Experience with failure-injection or replayable incident testing is beneficial
  • Experience building, integrating, evaluating, or orchestrating AI agents for infrastructure operations is welcome but not required; strong HPC operations and SRE skills are the foundation
  • Ability to understand production HPC infrastructure and learn unfamiliar systems quickly

Responsibilities:

  • Build and maintain a working understanding of our multi-cloud HPC architecture, including clusters, schedulers, accelerators, storage, networking, and authentication
  • Proactively identify reliability and performance issues by correlating logs, metrics, and events across cloud providers, clarifying incidents and uncovering telemetry gaps
  • Diagnose complex issues end to end across Linux hosts, containers, schedulers, distributed workloads, storage, networking, and accelerators, finding root causes rather than treating symptoms
  • Convert validated investigations into repeatable tools and agent-enabled workflows for detection, diagnosis, and bounded remediation
  • Build safety into automated changes through scoped permissions, precondition checks, human approval where needed, rollback, and post-change verification
  • Improve observability and monitoring, and assess automation using measures such as accuracy, recovery time, recurrence, and false positives
  • Partner with infrastructure, research, security, and cloud-provider teams to turn incident learnings into durable improvements

Technologies:

  • AI
  • AI Agents
  • AWS
  • Cloud
  • Grafana
  • Helm
  • InfiniBand
  • Kubernetes
  • Linux
  • NFS
  • OpenTelemetry
  • Prometheus
  • Python
  • RDMA
  • Security
  • Terraform

More:

We are seeking an engineer to help keep our growing multi-cloud AI infrastructure reliable and efficient. The role focuses on operational challenges across compute, accelerators, workload scheduling, storage, and networking, and on turning investigations into proactive detection, diagnosis, and safe automated remediation, including AI-powered agents. Prior experience building or orchestrating AI agents is not required.

last updated 46 week of 2026

Apply on DevITjobs

You apply on DevITjobs, where this job was posted. Peter Nowak isn’t the employer and doesn’t check listings.

Get new jobs like this by email

Other software · United States

Free. Confirm by email first; unsubscribe anytime in one click.

Similar jobs

Lead HPC Systems Engineer at EPAM Systems — New York, United States | Peter Nowak