Principal Software Engineer - CoreAI Model Inference & Serving | Microsoft Careers

Microsoft

Actively hiring

US Posted 70 days ago $139,900–$274,800 / year

View original post Log in to save

At a glance

AI generated

TL;DR

As a Principal Software Engineer in the CoreAI team at Microsoft, you will lead the development of the AI data-plane that powers large language model (LLM) inferencing workloads for Azure customers. Your day-to-day responsibilities include designing and implementing core serving systems, smart routing, and request distribution for various LLMs, ensuring reliability, scalability, and ultra-low latency. You will collaborate with world-class engineers to drive innovation in AI systems, conduct architecture reviews, mentor junior team members, and enhance the performance of mission-critical services. The role requires expertise in languages such as C++, Java, or similar, along with a deep understanding of distributed systems, system performance optimization, and engineering best practices for large-scale deployments. This position offers an unparalleled opportunity to shape the future of AI infrastructure at one of the fastest-growing services within Azure.

Skills

C++ Java Python Docker Kubernetes Terraform AWS Azure CI/CD PostgreSQL Redis Prometheus Grafana GraphQL REST gRPC OpenAPI Distributed Systems Scalability Reliability Observability

What you'll do

Design and implement core serving systems for various large language models (LLMs).
Develop smart routing and request distribution mechanisms for efficient service delivery.
Innovate in AI system architecture alongside top-tier engineering teams.
Enhance reliability, scalability, observability of mission-critical services through technical leadership.
Mentor junior engineers and conduct code reviews to uphold high coding standards.

What we're looking for

6+ years of technical engineering experience in C, C++, C#, or Java.
Extensive design and problem-solving skills with expertise in system performance and scalability.
Deep understanding of distributed systems for large-scale request serving and global infrastructure.
Proven track record of building high-quality, reliable systems at scale.
Experience with modern AI-assisted development tools to enhance engineering impact.
Customer-focused approach with a commitment to delivering impactful solutions.

Market check

Salary context

Competitive pay

How this pay compares to similar roles

Similar $196k

This role $207k

$124k most similar roles pay here $291k

This role pays more than 62% of similar roles. Most pay $177,250–$214,500 — the shaded band above. At the midpoint, this role pays about $207k versus about $196k for comparable roles.

Based on 240 similar postings.

Employer

About Microsoft

Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing

Microsoft currently has 534 open roles on FindRole.

Microsoft

US 98 days ago $139,900–$274,800

Python Java JavaScript Azure AWS GCP Kubernetes Docker CI/CD Prometheus Grafana Git A/B Testing Statistical Methodologies Security Hardening Threat Modeling Incident Response Observability Tools Logging Messaging Systems

Save