Principal Software Engineer, Distributed Systems Engineer

What you'll do

• Develop custom software for scheduling GPU resources on Kubernetes clusters.
• Implement monitoring and health management capabilities to ensure reliability of GPU assets.
• Process multiple data streams including hardware diagnostics, cluster telemetry, and network data.
• Maintain production AI clusters to ensure consistent performance and high availability.
• Evaluate system failures and improve services through a defined incident management process.
• Develop software using Kubernetes APIs and frameworks for large-scale distributed systems.
• Automate the management of large-scale infrastructure independent of cloud providers.

What we're looking for

• Experience in a software engineering role within a highly technical organization with demonstrable impact.
• Software development experience with Kubernetes APIs and frameworks beyond basic cluster operations.
• 15+ years of experience in similar roles and on large-scale production systems.
• Proficiency in a systems programming language such as Go or Python.
• Solid understanding of data structures and algorithms.
• Bachelor's degree in Computer Science, Engineering, Physics, Mathematics, or equivalent experience.
• Technical competency in managing and automating large-scale distributed systems independent of cloud providers (preferred).
• Advanced hands-on experience with cluster management systems like Kubernetes, Slurm, or Bright Cluster Manager (preferred).