Senior System Architect, Infrastructure Reliability

What you'll do

• Architect a scalable "flight recorder" framework to capture high-fidelity state across CPU, GPU, and Fabric during job failures.
• Build automated diagnostics to correlate GPU XID errors, PCIe bus failures, and CUDA memory exceptions with system-level events.
• Implement low-overhead tracing mechanisms to provide job execution visibility across multi-node Slurm or Kubernetes clusters.
• Develop machine learning models and heuristics to automatically classify failures as hardware, software, or environment issues.
• Define "signals of impending failure" with hardware teams to enable proactive job migration or checkpointing.
• Build high-performance daemons in C++ and Python to monitor system health without impacting workload performance.
• Utilize DCGM and NVML to monitor device health and capture state-dumps for GPU infrastructure.

What we're looking for

• BS, MS, or PhD in Computer Science or Electrical Engineering (or equivalent experience) with 6+ years in systems programming.
• Experience building automated RCA pipelines for HPC or cloud-scale environments.
• Expert knowledge of x86/ARM node-level metrics including IPC, cache contention, NUMA imbalance, and hardware interrupts.
• Strong C++ and Python skills to build high-performance daemons for monitoring system health.
• Familiarity with cluster resource managers such as Slurm, LSF, or Kubernetes.
• Expert knowledge of the Linux kernel and its error-reporting interfaces (preferred).
• Deep experience with NVIDIA DCGM and NVML for monitoring device health (preferred).
• Experience with checkpoint/restore technologies like CRIU (preferred).