Principal Software Engineer, AI Systems Observability

Microsoft

Confirmed live today High trust
Remote

Quick summary

Work type
Remote
Location
Remote
Salary
$142,800–$274,800 / yr
Posted
2 days ago
Freshness
Confirmed live today
Closes
Apr 6, 2027

Market check

Salary context

Competitive pay

How this pay compares to similar roles

Similar $215k
This role $209k
$127k $291k
below market most similar roles pay here above market

This role pays more than 56% of similar roles. Most pay $188,350–$242,524 — the blue band above. At the midpoint, this role pays about $209k versus about $215k for comparable roles.

Based on 240 similar postings.

Employer

About Microsoft

Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing

Microsoft currently has 648 open roles on FindRole.

Listed pay typically runs $119,800–$234,700 across 624 roles with salary data.

Most-posted roles

View all roles at Microsoft

At a glance

TL;DR · Principal Software Engineer, AI Systems Observability

As a Principal Software Engineer - AI Systems Observability, you will join the Azure Core Upstream Observability team to build open-source technologies for Kubernetes and Linux environments. You will lead the design and delivery of agentic observability capabilities, turning customer needs into coherent architectures and production-quality solutions. Your daily work involves mentoring engineers, performing hands-on implementation, and managing project plans across multiple groups. You will use C, C++, Go, Python, or Rust to collect and interpret systems signals using Linux kernel and extended Berkeley Packet Filter (eBPF) techniques. The role focuses on creating safe, observable, and evaluable agentic workflows by providing rich systems telemetry. You will solve complex problems regarding reliability and performance in cloud-scale distributed systems while supporting upstream communities and maintaining operational excellence.

What does a Software Engineer earn in Remote?

Median $208800 from 367 postings across 50 companies.

See salary data

What you'll do

  • Determine user requirements for agentic observability scenarios across Kubernetes and Linux environments.
  • Develop design documents for products and platforms that support safe, observable, and evaluable agentic workflows.
  • Perform hands-on implementation, code review, testing, debugging, and maintenance of extensible and maintainable code.
  • Collect, correlate, and interpret systems signals using Linux kernel and eBPF techniques.
  • Serve as a Designated Responsible Individual (DRI) to monitor systems and participate in on-call rotations.
  • Drive project plans, release plans, and work items across multiple groups and stakeholders.
  • Identify technical dependencies and patterns to improve system availability, reliability, and performance at scale.

What we're looking for

  • Bachelor's Degree in Computer Science or related technical field and 6+ years of technical engineering experience with coding in C, C++, Go, Python, or Rust.
  • Equivalent experience to the required degree and years of experience.
  • Ability to pass the Microsoft Cloud Background Check.
  • Master's Degree in Computer Science or related technical field and 8+ years of technical engineering experience with coding in C, C++, Go, Python, or Rust (preferred).
  • Bachelor's Degree in Computer Science or related technical field and 12+ years of technical engineering experience with coding in C, C++, Go, Python, or Rust (preferred).
  • Experience designing and operating at least one production Kubernetes/Linux observability or distributed-systems capability (preferred).
  • Experience building or operating at least one tool-using AI-agent system in a production-like environment with documented evaluation and safety controls (preferred).
  • Experience leading adoption of a shared technical architecture by at least two engineering teams or authoring a design accepted by an upstream open-source project (preferred).

More like this

Similar roles

Principal Software Engineer, AI Infrastructure

Microsoft

5 days ago $142,800–$274,800
Distributed Systems AI/ML Serving Azure Python C++ Java C# JavaScript C Observability Telemetry Distributed Tracing Capacity Management Load Balancing Reliability Engineering High-Performance Computing
6+ yrs exp Hybrid

Principal Software Engineer, Networking

Microsoft

Remote 4 days ago $142,800–$274,800
Kubernetes Go Rust C C++ Python Linux BGP eBPF WireGuard VXLAN GENEVE InfiniBand RDMA Cilium CI/CD Azure Containers GitHub GitLab
6+ yrs exp Remote

Senior Software Engineer, Agentic AI and Observability

Nvidia

Santa Clara, CA 33 days ago
Python Kubernetes Docker CI/CD Datadog OpenTelemetry Grafana Prometheus Terraform Pulumi LangChain LlamaIndex Semantic Kernel SRE DevOps GitOps Distributed Systems LLM Orchestration
8+ yrs exp

Principal AI Software Engineer

Palo Alto Networks

Santa Clara, CA 125 days ago $223,000–$306,500
Python Generative AI LLMs Multi-Agent Systems GPT-4 Llama 3 Anthropic LangSmith Arize HoneyHive OpenTelemetry Java Node.js Agile Scrum API Design AI Gateway Observability Pipelines
10+ yrs exp