Staff/ Principal Software Engineer, AI Platform Model Serving

What you'll do

• Lead the technical architecture and direction for a large-scale AI/ML model serving platform.
• Re-engineer GPU capacity provisioning to repurpose fleet resources for internal developer productivity.
• Implement intelligent routing, speculative decoding, and prompt caching to reduce inference costs.
• Build durable, resumable execution systems for long-running agentic workloads.
• Develop a unified serving layer for in-house infrastructure, third-party hosting, and external LLM providers.
• Optimize multi-GPU inference and disaggregated serving for high-traffic production environments.
• Shape long-term technical roadmaps to ensure scalability and business alignment for GenAI infrastructure.

What we're looking for

• Have a BS/MS/PhD in Computer Science, Applied Math, Engineering, or a related field.
• Have 12+ years of relevant experience in AI/ML infrastructure and distributed systems.
• Have experience with high-traffic distributed services and infrastructure for AI/ML model inference.
• Have experience building scalable model-serving solutions for Foundation Models with a focus on reducing latency and costs.
• Be familiar with deploying AI/ML models using NVIDIA Triton, vLLM, or SGLang, and scaling them via Kubernetes.
• Have experience working with public cloud infrastructure providers such as AWS, Azure, or GCP.
• Have open-source contributions to projects in the AI/ML, distributed systems, or infrastructure space (preferred).
• Have hands-on experience building AI models and exploring optimization techniques like quantization, compression, or pruning (preferred).