This role pays more than
78%
of similar roles. Most pay
$162,000–$229,193
— the shaded band above.
At the midpoint, this role pays about
$231k
versus about
$196k
for comparable roles.
Based on 240 similar postings.
Employer
About Microsoft
Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing
Microsoft currently has
622 open roles
on FindRole.
Listed pay typically runs
$119,800–$234,700
across 571 roles with salary data.
As an Applied Sciences IC6 at Core AI, you will join a pioneering team shaping the future of software development through Generative AI. Your primary responsibility is to develop and apply scientific methods for evaluating and measuring single-agent and multi-agent systems in production environments, focusing on quality, safety, reliability, cost, and behavioral consistency. You will design evaluation frameworks that integrate offline evaluations with real-time telemetry data to assess how changes impact agent performance, define quality benchmarks, build models for anomaly detection, and collaborate with engineering teams to operationalize these methods in production. Key technologies include Python, LangChain, OpenAI SDK, Azure Monitor, and distributed tracing systems. This role demands expertise in applied machine learning, statistical analysis, and observability data handling, as well as a background in AI safety and responsible AI practices.
Develop evaluation frameworks for single-agent and multi-agent systems to measure quality, safety, reliability, cost, and behavioral consistency.
Design methodologies linking offline evaluations, online signals, and production telemetry to assess real-world agent performance impacts.
Define scientifically grounded quality benchmarks for AI agents, including task success, tool-use effectiveness, plan quality, and user outcomes.
Build models and analysis techniques to detect regressions, identify root causes, and characterize diverse agent behaviors across environments.
Advance observability through new approaches like trace analysis, health modeling, behavioral clustering, anomaly detection, and multi-agent coordination.
What we're looking for
6+ years of experience in applied science, machine learning, or related technical fields.
Strong background in designing evaluation methodologies and measurement systems for complex intelligent systems.
Experience analyzing large-scale production data to drive product improvements.
Proficient coding skills in Python with a focus on working with engineering teams on production systems.
Advanced degree in Computer Science, Machine Learning, Statistics, Applied Mathematics, or related field.
Familiarity with agent frameworks and evaluation frameworks for AI systems.
Background in AI safety, guardrails, and responsible AI measurement.
Python
Java
C++
TensorFlow
PyTorch
Kubernetes
Docker
AWS
Azure
Google Cloud Platform
PostgreSQL
MongoDB
CI/CD
Git
GitHub
Jenkins
Prometheus
Grafana
Elasticsearch
Kafka
Spark
Hadoop
Distributed Systems
Large Language Models
Information Retrieval
Learning-to-Rank
Conversational AI
Agentic AI
TensorFlow
PyTorch
LLM training
Artificial Intelligence
Data Science
Information Retrieval
Machine Learning
Natural Language Processing
Big Data
Spark
MapReduce
Cosmos
Search
Recommendations
CI/CD