As a Principal Software Engineer on the Artificial Intelligence Cloud Inference team at Microsoft, you will work closely with OpenAI to optimize the performance of large language models (LLMs) across various hardware platforms, including GPUs and custom silicon. Your daily tasks will involve benchmarking LLMs for performance, debugging and optimizing code at multiple levels, from kernels to systems, and developing software tools to enhance insights into performance bottlenecks. You will also collaborate with internal and external partners to ensure fast deployment of models on new hardware and contribute to achieving Azure's capital expenditure goals by reducing the computing fleet footprint. This role requires expertise in high-performance applications, GPU architecture, and deep learning frameworks like PyTorch or TensorFlow, as well as proficiency in CUDA and ROCm for efficient model inference and optimization.
Skills
Python
C++
CUDA
ROCm
PyTorch
TensorFlow
ONNX_Runtime
NVIDIA_GPUs
AMD_GPUs
Azure
CI/CD
GPU_Profiling_Tools
LLMs
DNN_Inference
Performance_Optimization
High_Performance_Computing
What you'll do
Identify and drive improvements to end-to-end inference performance of OpenAI and other LLM models.
Benchmark performance on Nvidia/AMD GPUs and Microsoft silicon for various AI models.
Optimize and monitor performance at multiple levels including model, algorithm, system, and kernel.
Build software tooling to enable insights into performance opportunities across different abstraction layers.
Design, implement, and test functions or components for AI/DNN/LLM frameworks and tools.
Reduce complexity of key components/pipelines to improve overall system efficiency.
What we're looking for
4+ years of practical experience working on high-performance applications.
Solid foundation in software engineering principles, computer architecture, and GPU architecture.
Experience in end-to-end performance analysis and optimization of state-of-the-art LLMs and HPC applications.
Proficiency using GPU profiling tools such as CUDA or ROCm.
Hands-on experience with DNN/LLM inference and familiarity with DL frameworks like PyTorch or TensorFlow.
Market check
Salary context
This
$139,900–$274,800
range sits above
62%
of similar postings on FindRole.
Peer median band
$140,600–$261,850
Median floor and ceiling across peers.
Typical midpoint (25–75%)
$177,250–$214,625
Middle half of comparable postings.
Based on 240 comparable postings.*
* 240 is the maximum number of comparable postings sampled.
Employer
About Microsoft
Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing
Microsoft currently has
451 open roles
on FindRole.
Listed pay typically runs
$119,800–$234,700
across 417 roles with salary data.