Senior Software Engineer, Private Cloud Compute Inference Routing

What you'll do

• Design and operate inference routing and request scheduling services for global traffic management.
• Develop adaptive routing strategies to balance latency, availability, and compute capacity.
• Build admission control, prioritization, and overload management mechanisms for competing workloads.
• Implement routing strategies for prefill/decode disaggregation and cache locality within a privacy architecture.
• Design network protocols and write high-performance server software for request dispatch and streaming.
• Benchmark routing strategies using queueing delay, throughput, and fleet utilization metrics.
• Own technical decisions from early architecture through production rollout for new inference capabilities.

What we're looking for

• 3+ years of professional experience designing, building and operating large distributed systems in production.
• Proficiency in server development using Java, C/C++, C#, Go, or Swift.
• Knowledge of LLM inference serving, including request routing, KV-cache placement, and prefill/decode disaggregation.
• Experience with network protocols, application-level networking, and client-server architectures for load balancing or scheduling at scale.
• Experience benchmarking, debugging, and tuning distributed systems with knowledge of Unix system internals.
• Experience with cloud deployment and observability technologies such as Kubernetes.
• Bachelor's degree in an engineering related field or equivalent experience.
• Experience building LLM inference serving or routing systems such as llm-d, NVIDIA Dynamo, vLLM, or SGLang (preferred).