Senior System Software Engineer, Agentic Inference

Nvidia

Confirmed live today High trust

Hybrid

Quick summary

Work type: Hybrid
Location: Santa Clara, CA
Salary: $224,000–$356,500 / yr
Employment: Full-time
Posted: 92 days ago
Freshness: Confirmed live today
Nearby: 99+ roles within 25 mi

Market check

Salary context

Above market

How this pay compares to similar roles

Similar $206k

This role $290k

$145k $379k

below market most similar roles pay here above market

This role pays more than 92% of similar roles. Most pay $177,250–$235,750 — the blue band above. At the midpoint, this role pays about $290k versus about $206k for comparable roles.

Based on 240 similar postings.

Employer

About Nvidia

Nvidia is a leading designer of graphics processing units (GPUs) and system-on-chip units, powering gaming, professional visualization, data centers, and artificial intelligence workloads. Industry: Semiconductors & AI Computing

Nvidia currently has 1623 open roles on FindRole.

Listed pay typically runs $184,000–$287,500 across 1110 roles with salary data.

Most-posted roles

View all roles at Nvidia

At a glance

TL;DR · Senior System Software Engineer, Agentic Inference

Role Posting Log in to save

Senior System Software Engineer, Agentic Inference - Dynamo will join the GPU-accelerated deep learning software team to develop open source software for serving inference of trained AI models on GPUs. The role involves contributing to disaggregated serving for Dynamo-supported inference engines including vLLM, SGLang, and TRT-LLM while expanding capabilities for agentic inference workloads like long-horizon reasoning and tool calling. Key responsibilities include innovating in inference-state management for KV- and prefix-cache reuse across heterogeneous memory hierarchies with NIXL to improve latency and throughput. Candidates must possess excellent Rust and Python programming skills, performance analysis experience, and an understanding of modern LLM API semantics. The work focuses on building a distributed inference frontend to support new models and stateful Responses API semantics for self-hosted LLMs.

What does a System Software Engineer earn in California?

Median $235750 from 70 postings across 4 companies.

See salary data

Skills

Rust Python vLLM SGLang TensorRT-LLM GPU Deep Learning Generative AI Prefix Cache NIXL LLM Performance Analysis Debugging

What you'll do

Develop open source software to serve inference of trained AI models on GPUs.
Contribute to disaggregated serving for Dynamo-supported inference engines like vLLM and SGLang.
Expand inference capabilities to support agentic workloads including long-horizon reasoning and tool calling.
Innovate in inference-state management for KV-cache and prefix-cache reuse across memory hierarchies.
Build and evolve the distributed inference frontend to provide day-0 support for new models.
Load-balance asynchronous requests across available resources while optimizing throughput under latency constraints.
Integrate the latest open source technology to build robust and scalable high-performance software components.

What we're looking for

Masters or PhD or equivalent experience in Computer Science, Computer Engineering, or a related field.
10+ years of experience in Computer Science, Computer Engineering, or a related field.
Excellent Rust and Python programming skills, including software design, debugging, performance analysis, and test design.
Understanding of modern LLM API semantics, including structured outputs, tool calling, reasoning controls, and context management.
Ability to work in a fast-paced, agile team environment.
Prior contributions to open-source AI inference frameworks such as vLLM, TensorRT-LLM, or SGLang (preferred).
Experience optimizing GPU memory, KV/prefix caches, or high-performance networking for long-context and reasoning workloads (preferred).
Experience integrating self-hosted LLM serving stacks with agent harnesses like OpenCode, Codex, Claude Code, or Pi (preferred).