This role pays more than
57%
of similar roles. Most pay
$177,250–$208,800
— the shaded band above.
At the midpoint, this role pays about
$177k
versus about
$193k
for comparable roles.
Based on 239 similar postings.
Employer
About Microsoft
Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing
Microsoft currently has
1559 open roles
on FindRole.
Listed pay typically runs
$119,800–$234,700
across 1390 roles with salary data.
Join the Core AI team as a Senior or Principal Applied Scientist, where you will drive the development of scientific methodologies to evaluate and measure single-agent and multi-agent systems in production. Your daily tasks include creating evaluation frameworks that assess quality, safety, reliability, cost, and behavioral consistency, while also designing methods to connect offline evaluations with real-world agent performance through online signals and telemetry data. You’ll work on defining quality benchmarks, building models for anomaly detection, and collaborating with engineering teams to operationalize these systems in production. The role requires expertise in Python or similar languages, experience with AI safety and responsible AI practices, familiarity with evaluation frameworks like LangChain and OpenAI SDK, and a background in machine learning and statistical methods. This position is pivotal in shaping how Microsoft measures and improves observable AI systems at scale.
Develop evaluation frameworks for single-agent and multi-agent systems to measure quality, safety, reliability, cost, and behavioral consistency.
Design methodologies linking offline evaluations, online signals, and production telemetry to assess real-world agent performance impacts.
Define scientifically grounded benchmarks for AI agents, including task success, tool-use effectiveness, plan quality, failure modes, and user outcomes.
Build models and techniques to detect regressions, identify root causes, and characterize diverse agent behaviors across workflows.
Advance observability through new approaches like trace analysis, health modeling, behavioral clustering, anomaly detection, and multi-agent coordination.
What we're looking for
Advanced degree in Computer Science, Machine Learning, Statistics, Applied Mathematics, or related field.
6+ years of experience designing evaluation methodologies, experiments, or measurement systems for complex intelligent or distributed systems.
Strong coding skills in Python or similar languages and ability to work with engineering teams on production-facing systems.
Experience building or evaluating large-scale LLM- or agent-based systems in production environments.
Familiarity with AI safety, guardrails, responsible AI measurement, and evaluation frameworks for AI systems.
Background in telemetry analysis, distributed tracing data, and observability systems in large-scale environments.