Senior System Software Engineer, Agentic Inference

What you'll do

• Develop open source software to serve inference of trained AI models running on GPUs.
• Contribute to disaggregated serving for Dynamo-supported engines like vLLM, SGLang, and TRT-LLM.
• Expand capabilities to support agentic inference workloads including long-horizon reasoning and tool calling.
• Innovate in inference-state management using KV-cache and prefix-cache reuse across heterogeneous memory hierarchies.
• Build and evolve Dynamo’s distributed inference frontend for new models and upstream API compatibility.
• Load-balance asynchronous requests across available resources to optimize throughput under latency constraints.
• Integrate the latest open source technologies into the generative AI inference platform.

What we're looking for

• Master's degree, PhD, or equivalent experience in Computer Science, Computer Engineering, or a related field.
• 10+ years of experience in Computer Science, Computer Engineering, or a related field.
• Proficiency in Rust and Python programming for software design, debugging, performance analysis, and test design.
• Understanding of modern LLM API semantics including structured outputs, tool calling, reasoning controls, and context management.
• Experience contributing to open-source AI inference frameworks such as vLLM, TensorRT-LLM, or SGLang.
• Experience optimizing GPU memory, KV/prefix caches, or high-performance networking for long-context workloads.
• Understanding of LLM-specific inference challenges for agentic workloads and multi-turn state reuse.
• Experience integrating self-hosted LLM serving stacks with agent harnesses like OpenCode, Codex, Claude Code, or Pi.