below marketmost similar roles pay hereabove market
This listing doesn't post a salary. Most similar roles pay
$202,725–$267,050.
Based on 240 similar postings.
Employer
About Nvidia
Nvidia is a leading designer of graphics processing units (GPUs) and system-on-chip units, powering gaming, professional visualization, data centers, and artificial intelligence workloads. Industry: Semiconductors & AI Computing
Nvidia currently has
1623 open roles
on FindRole.
Listed pay typically runs
$184,000–$287,500
across 1110 roles with salary data.
Engineering Manager, LLM Inference & Deployment at Scale leads a team responsible for deploying and serving Large Language Models and Vision-Language Models at scale. The role involves managing engineers to transition modern models from research into optimized, reliable, and scalable production deployments. Key responsibilities include driving technical strategy for low-latency, high-throughput, and cost-efficient inference while establishing engineering standards for benchmarking and profiling. The position requires expertise in inference optimization techniques like quantization, speculative decoding, continuous batching, and KV-cache optimization. Candidates must understand distributed inference, multi-node deployments, and GPU cluster orchestration. The work involves analyzing deep learning workloads across the inference stack and hardware, utilizing tools such as TensorRT, TensorRT-LLM, vLLM, and SGLang to solve production challenges for rapidly evolving generative AI models.
What does a Engineering Manager earn in California?
Median $279400 from 84 postings across 24 companies.
Lead and mentor a high-performing engineering team to build and operate a GenAI model deployment platform.
Drive the deployment and optimization of LLMs and VLMs for low-latency, high-throughput, and cost-efficient inference.
Analyze and optimize end-to-end deep learning workloads across the model stack, distributed systems, and GPU hardware.
Transition new model architectures and prototypes from research teams into production-ready deployments.
Define the technical strategy and roadmap for model deployment, scalability, reliability, and performance.
Establish engineering standards for benchmarking, profiling, and continuous performance optimization.
Manage the deployment of rapidly evolving GenAI models in collaboration with internal and external partners.
What we're looking for
Master’s or PhD in Computer Science, Computer Engineering, or a related field, or equivalent experience.
8+ years of overall experience.
3 years of management or leadership experience.
Strong hands-on experience with LLMs and/or VLMs and a solid understanding of modern deep learning architectures.
Deep expertise in inference optimization techniques such as quantization, speculative decoding, continuous batching, prefix caching, and KV-cache optimization.
Experience with disaggregated inference/serving, distributed inference, multi-node deployments, and GPU cluster orchestration.
Strong technical leadership, communication, and people-management skills.
Experience building or operating AI inference, model serving, or model deployment platforms (preferred).
CUDA
Python
C++
Rust
TensorRT-LLM
vLLM
SGLang
Triton
PyTorch
NVIDIA Nsight Systems
CUTLASS
NCCL
Quantization
Distributed Systems
GPU Architecture
Model Parallelism
Speculative Decoding