Senior Software Engineer - CoreAI Model Inference & Serving | Microsoft Careers

Microsoft

Actively hiring
San Francisco Bay area · New York City metropolitan area Posted 69 days ago $119,800–$234,700 / year
View original post Log in to save

At a glance

AI generated

TL;DR

As a Senior Software Engineer in the CoreAI team at Microsoft Azure, you will play a pivotal role in developing the AI data-plane that powers large language model (LLM) inferencing workloads for diverse clients. Your responsibilities include designing and implementing core serving systems, smart routing, and request distribution for various LLMs, ensuring reliability, scalability, and ultra-low latency. You will leverage C++, Java, or similar languages to build large-scale AI services and platform capabilities that drive new product experiences. This role requires expertise in distributed systems, system performance optimization, and modern AI-assisted development tools, with a focus on delivering high-quality solutions at scale for Microsoft’s expansive catalog of LLMs.

Skills

C++ Java Python Docker Kubernetes Terraform AWS Azure CI/CD Prometheus Grafana PostgreSQL Redis MongoDB Distributed Systems Microservices RESTful APIs GraphQL Observability Scalability Reliability

What you'll do

  • Design and implement core serving systems for large language models (LLMs).
  • Develop smart routing and request distribution mechanisms for diverse LLMs.
  • Build scalable AI services to power new products and customer experiences.
  • Drive innovation in AI system architecture and engineering best practices.
  • Enhance reliability, scalability, observability of mission-critical AI services.
  • Mentor team members through code reviews and technical guidance.

What we're looking for

  • 4+ years of experience in designing and coding large-scale AI services and platforms.
  • Proficient in C, C++, C#, or Java for developing core serving systems and smart routing.
  • Expertise in building high-quality, reliable systems with a focus on scalability and performance.
  • Understanding of distributed systems and request serving at scale across global infrastructure.
  • Experience using modern AI-assisted development tools to enhance engineering efficiency and quality.
  • Hands-on technical leadership in architecture design, code reviews, and mentorship.

Market check

Salary context

This $119,800–$234,700 range sits above 72% of similar postings on FindRole.

Peer median band

$119,800–$234,000

Median floor and ceiling across peers.

Typical midpoint (25–75%)

$149,800–$196,750

Middle half of comparable postings.

Based on 240 comparable postings.

* 240 is the maximum number of comparable postings sampled.

Employer

About Microsoft

Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing

Microsoft currently has 445 open roles on FindRole.

Listed pay typically runs $119,800–$234,700 across 415 roles with salary data.

Most-posted roles

View all roles at Microsoft

More like this

Similar roles

Principal Software Engineer, CoreAI | Microsoft Careers

Microsoft

US 79 days ago $139,900–$274,800
Python C++ Java JavaScript Azure CI/CD Kubernetes Docker Terraform Prometheus Grafana LLMs SLMs Multimodal_Models Code_Specific_Models Scalability Reliability Security Privacy Cloud_Infrastructure DevOps