Senior Software Engineer, CoreAI Workload Engines

Microsoft

Quick summary

Work type: On-site
Location: —
Salary: $119,800–$234,700 / yr
Posted: 81 days ago
Closes: Oct 4, 2026

Market check

Salary context

Competitive pay

How this pay compares to similar roles

Similar $193k

This role $177k

$106k most similar roles pay here $248k

This role pays more than 51% of similar roles. Most pay $163,500–$222,000 — the shaded band above. At the midpoint, this role pays about $177k versus about $193k for comparable roles.

Based on 240 similar postings.

Employer

About Microsoft

Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing

Microsoft currently has 622 open roles on FindRole.

Listed pay typically runs $119,800–$234,700 across 571 roles with salary data.

Most-posted roles

View all roles at Microsoft

At a glance

TL;DR · Senior Software Engineer, CoreAI Workload Engines

Role Posting Log in to save

The CoreAI Workloads team at Azure seeks a Senior engineer to build foundational inference engines and APIs for large-scale AI systems. This role involves optimizing OpenAI and open-source models by implementing performance improvements, running end-to-end experiments, and building experimentation capabilities to ensure safe and repeatable changes. You will work on the AI software stack, including runtime, scheduling, and serving paths, focusing on latency, throughput, availability, and cost efficiency. Key responsibilities include designing scalable inference architectures, extending infrastructure abstractions for elastic engines, tuning GPU performance, and collaborating with networking teams for high-performance interconnects. The ideal candidate has experience in C++, Python, Kubernetes, and optimizing large language models, along with strong skills in benchmarking, profiling, and cross-layer debugging.

Skills

Python Kubernetes PyTorch CUDA Prometheus Grafana CI/CD Docker PostgreSQL Redis OpenAI Azure NVIDIA GPUs RDMA InfiniBand RoCE NCCL TensorFlow Hadoop Apache Spark GitLab Jenkins

What you'll do

Optimize inference engines for OpenAI and open-source models by implementing performance improvements.
Run end-to-end experiments to measure and improve latency, throughput, availability, and cost of AI workloads.
Build experimentation capabilities for large-scale AI inference to ensure quick and safe iterative development.
Own serving availability and efficiency for Azure OpenAI Service through tiered experimentation and multi-modal utilization.
Design and evolve inference serving architectures using techniques like disaggregated serving and quantization for improved performance.

What we're looking for

Proven ability to design and operate large-scale production inference services.
Strong skills in performance analysis including benchmarking, profiling, diagnosing regressions.
Hands-on experience optimizing LLM inference in production environments.
Experience with Kubernetes for building and operating services on k8s platforms.
Demonstrated technical leadership and cross-team architectural alignment capabilities.
Familiarity with GPU-accelerated inference stacks and high-performance networking.

Save