below marketmost similar roles pay hereabove market
This role pays more than
99%
of similar roles. Most pay
$196,000–$255,125
— the blue band above.
At the midpoint, this role pays about
$352k
versus about
$226k
for comparable roles.
Based on 240 similar postings.
Employer
About Nvidia
Nvidia is a leading designer of graphics processing units (GPUs) and system-on-chip units, powering gaming, professional visualization, data centers, and artificial intelligence workloads. Industry: Semiconductors & AI Computing
Nvidia currently has
1792 open roles
on FindRole.
Listed pay typically runs
$184,000–$287,500
across 1179 roles with salary data.
The Principal High-Performance LLM Training Engineer joins the team to drive performance for large-scale AI training and post-training workloads across the full hardware and software stack. This role involves leading end-to-end performance analysis and optimization of frontier-scale LLM workloads running on thousands of GPUs. You will develop production-quality software, tools, models, and benchmarks to improve training efficiency and developer velocity. Key responsibilities include identifying bottlenecks in compute, memory, communication, and scheduling while refining performance models to guide future architecture decisions. You will utilize PyTorch, JAX, NeMo, and NeMo RL alongside CUDA libraries, compilers, and networking. The work focuses on solving technical challenges in distributed training, including data, tensor, pipeline, and expert parallelism for transformer-based models and reinforcement learning.
Lead end-to-end performance analysis and optimization of LLM pre-training and post-training workloads on NVIDIA hardware.
Identify and remove bottlenecks across compute, memory, communication, scheduling, and framework overhead to reach speed-of-light performance.
Develop production-quality software, tools, models, and benchmarks to improve training performance and developer velocity.
Build and refine performance models and simulation methodologies to guide future GPU, networking, and software architecture decisions.
Translate workload insights into concrete hardware and software recommendations for the broader AI ecosystem.
Serve as a technical authority by partnering with teams across GPU architecture, CUDA libraries, compilers, and networking.
Mentor engineers and establish best practices for large-scale AI performance analysis and optimization.
What we're looking for
MS or PhD in Computer Science, Electrical Engineering, Computer Engineering, or a related field, or equivalent experience.
12+ years of relevant work or research experience.
Demonstrated principal-level technical impact in large-scale AI training systems, GPU performance optimization, distributed systems, HPC, ML frameworks, compilers, or hardware/software co-design.
Deep hands-on experience analyzing and optimizing performance of large-scale deep learning workloads, especially transformer-based models and LLM pre-training.
Strong understanding of GPU and AI accelerator architecture from individual accelerators to datacenter-scale systems.
Experience with distributed training techniques including data, tensor, pipeline, expert, and sequence parallelism.
Strong track record of using profiling, tracing, benchmarking, and performance modeling tools to diagnose complex bottlenecks.
Excellent communication and technical leadership skills to influence architecture and software decisions across multiple teams.
PyTorch
JAX
CUDA
Python
C++
GPU Architecture
Deep Learning
Neural Networks
Performance Analysis
Performance Modeling
LLM Training
Computer Architecture
Simulators
PyTorch
JAX
CUDA
C++
Python
GPU Computing
Deep Learning
Neural Networks
Computer Architecture
MLPerf
Simulators
System-level Performance
AI Training Workloads
Remote (Santa Clara, CA)+45 days ago
$224,000–$356,500
C++
Python
CUDA
PyTorch
JAX
XLA
GPU Computing
Distributed Systems
Performance Engineering
Profiling
Benchmarking
Operating Systems
Computer Architecture
High-Performance Computing
Data Analysis
Observability
PyTorch
JAX
TensorFlow
Python
C++
CUDA
Kubernetes
SLURM
MLOps
Distributed Training
GPU Acceleration
HPC
Data Loaders
Model Architecture
Foundation Models
LLM