Principal Software Engineer, CoreAI Workload Engines | Microsoft Careers

Microsoft

Actively hiring
San Francisco Bay area · New York City metropolitan area Posted 52 days ago $142,800–$274,800 / year
View original post Log in to save

At a glance

AI generated

TL;DR

As a Principal Engineer on the CoreAI Workloads team at Azure, you will lead the optimization of inference engines for OpenAI and open-source models, focusing on performance improvements across runtime, scheduling, and serving paths. Your daily tasks include running end-to-end experiments to measure and improve latency, throughput, availability, and cost, while also building experimentation capabilities that ensure safe and repeatable changes. You’ll work closely with the data plane, compute, and partner teams to enhance inference serving architectures using techniques like disaggregated serving and multi-token prediction. Essential skills include experience in performance analysis, debugging complex systems issues, and hands-on expertise with Kubernetes for production services. Preferred qualifications involve optimizing LLM inference, familiarity with GPU-accelerated stacks, and building experimentation systems. This role offers the chance to influence cloud GPU platforms used by Fortune 500 enterprises and startups alike, collaborating with experts across various technical domains.

Skills

Python Kubernetes PyTorch CUDA Prometheus Grafana CI/CD Docker PostgreSQL Redis OpenAI LLM Azure NVIDIA GPUs RDMA InfiniBand RoCE NCCL TensorFlow C++ Java JavaScript Go Git Jenkins Ansible Terraform AWS Google Cloud Platform CI/CD pipelines

What you'll do

  • Optimize inference engines for OpenAI and open-source models by implementing performance improvements.
  • Run end-to-end experiments to measure and improve latency, throughput, availability, and cost of AI services.
  • Build experimentation capabilities for large-scale AI inference to ensure quick and safe iteration.
  • Own serving availability and efficiency for Azure OpenAI Service workloads through tiered experimentation and multi-modal utilization.
  • Design and evolve inference serving architectures to enhance utilization and reduce latency using advanced techniques.

What we're looking for

  • Proven ability to design and operate large-scale production inference services.
  • Strong skills in performance analysis including benchmarking, profiling, diagnosing regressions.
  • Hands-on experience with Kubernetes for building and operating services.
  • Demonstrated technical leadership and mentoring engineers across teams.
  • Experience optimizing LLM inference in production environments (preferred).
  • Familiarity with GPU-accelerated inference stacks and high-performance networking.

Market check

Salary context

This $142,800–$274,800 range sits above 68% of similar postings on FindRole.

Peer median band

$139,900–$239,250

Median floor and ceiling across peers.

Typical midpoint (25–75%)

$177,250–$214,500

Middle half of comparable postings.

Based on 240 comparable postings.

* 240 is the maximum number of comparable postings sampled.

Employer

About Microsoft

Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing

Microsoft currently has 451 open roles on FindRole.

Listed pay typically runs $119,800–$234,700 across 417 roles with salary data.

Most-posted roles

View all roles at Microsoft

More like this

Similar roles

Principal Software Engineer, CoreAI | Microsoft Careers

Microsoft

US 67 days ago $139,900–$274,800
C++ Kubernetes CUDA Docker Azure Linux Performance Profiling Tools Debugging Tools CI/CD Multimodal Inferencing LLM Inferencing Infrastructure Service Reliability Engineering OpenAI

Principal Software Engineer, CoreAI | Microsoft Careers

Microsoft

US 72 days ago $142,800–$274,800
Kubernetes Python C C++ Java JavaScript Terraform AWS Azure PostgreSQL CI/CD Prometheus Grafana Docker RDMA InfiniBand NCCL CUDA AKS Dynamic Resource Allocation(DRA)

Principal Software Engineer, CoreAI | Microsoft Careers

Microsoft

US 77 days ago $139,900–$274,800
Python C++ Java JavaScript Azure CI/CD Kubernetes Docker Terraform Prometheus Grafana LLMs SLMs Multimodal_Models Code_Specific_Models Scalability Reliability Security Privacy Cloud_Infrastructure DevOps