Principal Software Engineer, Distributed Systems

What you'll do

• Develop custom software to schedule GPU resources on Kubernetes clusters.
• Implement monitoring and health management capabilities for GPU assets.
• Harness multiple data streams including hardware diagnostics and network telemetry.
• Ensure production AI clusters run reliably and consistently with maximum performance.
• Evaluate system failures and improve services using a defined incident management process.
• Develop and maintain large-scale distributed systems independent of cloud providers.
• Automate cluster management systems including Kubernetes, Slurm, and Bright Cluster Manager.

What we're looking for

• 15+ years of experience in a similar role and experience on large-scale production systems.
• Direct experience in a software engineering role within a highly technical organization with demonstrable impact.
• Software development experience with Kubernetes APIs and frameworks, including cluster operations and operator development.
• Technical knowledge of a systems programming language like Go or Python and a solid understanding of data structures and algorithms.
• Experience with GPU resource scheduling and implementing monitoring and health management capabilities for GPU assets.
• BS in Computer Science, Engineering, Physics, Mathematics, or a comparable degree or equivalent experience.
• Technical competency in managing and automating large-scale distributed systems independent of cloud providers (preferred).
• Advanced hands-on experience and deep understanding of cluster management systems like Kubernetes, Slurm, or Bright Cluster Manager (preferred).