Senior Data Scientist, Copilot Evals

Microsoft

Confirmed live 2 days ago High trust

Quick summary

Work type
On-site
Location
Redmond, WA
Salary
$119,800–$234,700 / yr
Posted
16 days ago
Freshness
Confirmed live 2 days ago
Closes
Feb 22, 2027

Market check

Salary context

Competitive pay

How this pay compares to similar roles

Similar $181k
This role $177k
$106k most similar roles pay here $248k

This role pays more than 56% of similar roles. Most pay $147,775–$214,800 — the shaded band above. At the midpoint, this role pays about $177k versus about $181k for comparable roles.

Based on 240 similar postings.

Employer

About Microsoft

Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing

Microsoft currently has 598 open roles on FindRole.

Listed pay typically runs $119,800–$234,700 across 586 roles with salary data.

Most-posted roles

View all roles at Microsoft

At a glance

TL;DR · Senior Data Scientist, Copilot Evals

The Senior Data Scientist for Copilot Evals joins the CADET team to build a customer-grounded quality system for Copilot. This role focuses on creating measurement and decision systems that determine where to invest resources based on customer scenarios, intents, and usage patterns. You will develop data-driven prioritization frameworks, define taxonomies for loss patterns and coverage, and design quality gates to ensure products reflect real user experiences rather than static benchmarks. Key responsibilities include analyzing offline-online agreement, designing sampling and clustering methods for noisy signals, and using causal and experimental methods to inform product investments. The role requires expertise in SQL, Python or R, and statistical analysis techniques like experimentation and segmentation. You will work with complex data to solve problems regarding AI product quality, LLM evaluation, and translating customer evidence into reward signals and automated pipelines.

What does a Data Scientist earn?

Median $162025 from 276 postings across 61 companies.

See salary data

What you'll do

  • Build a data-driven prioritization framework to guide quality investments based on customer impact, usage, and severity.
  • Define and maintain taxonomies for intents, customer scenarios, and loss patterns across triage and reporting systems.
  • Measure how well evaluation portfolios represent production traffic, customer segments, and various failure modes.
  • Design quality gates with specific success thresholds, run cadences, and escalation criteria for top customers.
  • Create recurring scorecards to connect offline evaluation metrics with online measures like DSAT and task completion.
  • Develop advanced sampling, clustering, and trend-detection methods to process noisy customer and product signals.
  • Use causal and experimental methods to translate data findings into concrete product and model investment decisions.
  • Translate customer evidence into valid task distributions, reward signals, and automated reporting pipelines.

What we're looking for

  • A Doctorate in a relevant field and 1+ years of data science experience.
  • A Master's degree in a relevant field and 3+ years of data science experience.
  • A Bachelor's degree in a relevant field and 5+ years of data science experience.
  • Equivalent experience to the specified educational and experience requirements.
  • Expertise in SQL and at least one analytical programming language such as Python or R (preferred).
  • Solid foundation in statistical analysis, experimentation, sampling, segmentation, measurement, and data visualization (preferred).
  • Experience integrating noisy quantitative and qualitative signals into actionable prioritization or measurement frameworks (preferred).
  • Experience with AI product quality, LLM evaluation, DSAT, or model telemetry (preferred).

More like this

Similar roles

Senior Data Scientist

Intuit

San Diego, CA +1 92 days ago $162,500$220,000
Machine Learning Causal Inference Python SQL R Generative AI Deep Learning Time-series Forecasting Uplift Modeling A/B Testing Bayesian Inference Tableau Qlik Experimental Design Predictive Modeling Econometrics Data Visualization
4+ yrs exp

Senior Data Scientist

Mastercard

San Francisco, CA 5 days ago $138,000$221,000
Machine Learning Python Apache Spark AWS Azure GCP MLOps A/B Testing Causal Inference RTB AdTech Statistics Predictive Modeling Feature Engineering Real-time Pipelines

Senior Data Scientist

Q2

Cary, NC 28 days ago
Python R SQL Machine Learning LLM RAG Prompt Engineering Git Vector Databases AWS GCP Azure PowerBI Data Science Anomaly Detection Embedding-based Retrieval
8+ yrs exp

Senior Data Scientist

Alloy

New York, NY 157 days ago $212,000$251,000
Machine Learning Python SQL Feature Engineering Tree-based Models Looker Model Evaluation Production ML Workflows Dashboards Fraud Detection
8+ yrs exp Hybrid

Senior Data Scientist

Daily Pay

New York, NY 99 days ago $136,000$209,000
Machine Learning Python SQL Feature Engineering Data Pipelines Model Monitoring Drift Detection Statistics AI
5+ yrs exp

Senior Data Scientist

General Dynamics

Sterling, VA 73 days ago $152,150$205,850
Python Machine Learning AI Data Modeling Language Models SQL Statistics Linear Algebra Calculus Linux Windows
10+ yrs exp Hybrid