Engineering Manager, Private Cloud Compute Inference Routing

What you'll do

• Lead the design and execution of global routing and scheduling strategies for LLM inference requests.
• Manage the development of an inference AI gateway for authentication, rate limiting, and streaming responses.
• Establish technical strategies for capacity-aware placement, load balancing, and failover across a multi-region fleet.
• Oversee the delivery of adaptive routing, admission control, and overload management to maintain service responsiveness.
• Evaluate and implement cache-aware request placement and prefill/decode disaggregation within a privacy-preserving architecture.
• Scale operational practices including observability, service-level objectives, testing, and incident management.
• Drive improvements in routing latency, availability, and fleet utilization through benchmarking and capacity-aware scheduling.
• Recruit, develop, and lead a high-performing organization of engineers and engineering managers.

What we're looking for

• Extensive software engineering experience leading teams that design, build, and operate large-scale distributed systems in production.
• Proven ability to recruit, develop, and retain high-performing engineers and leaders.
• Experience delivering API gateways, request routing, load balancing, scheduling, or traffic management systems at scale.
• Expertise in LLM inference serving, including request routing, KV-cache placement, and prefill/decode disaggregation.
• Record of shipping highly reliable systems in high-availability environments, including leading incident management and on-call practices.
• Technical depth in server development with experience in Java, C/C++, Go, or Swift.
• Bachelor’s degree in an engineering discipline or equivalent professional experience.
• Proven leadership in developing LLM inference serving or routing platforms (preferred).