Senior Data Scientist

What you'll do

• Design and execute analytical frameworks to evaluate product investments across Outlook Mail, Calendar, Search, and Copilot.
• Define key metrics, guardrails, and success criteria for features in collaboration with product and engineering teams.
• Design and analyze online experiments, quasi-experiments, and observational studies to determine causal impact on product decisions.
• Develop dashboards and reporting pipelines to monitor product health, AI quality, reliability, and user adoption.
• Evaluate generative AI experiences using offline/online methods including benchmark datasets, human judgment, and LLM-as-a-judge techniques.
• Apply prompt engineering and evaluation techniques to improve the safety, relevance, and accuracy of AI responses.
• Analyze large-scale behavioral and telemetry data to identify customer needs and forecast trends across segments.
• Translate complex data into actionable insights for senior leadership to influence product strategy and roadmap decisions.

What we're looking for

• A Doctorate, Master's, or Bachelor's degree in a quantitative field (e.g., Data Science, Math, CS) with 1, 3, or 5 years of experience respectively.
• Ability to pass the Microsoft Cloud background check.
• Experience with Generative AI technologies including LLMs, RAG, agentic systems, and NLP (preferred).
• Hands-on experience in AI evaluation, prompt engineering, and A/B testing (preferred).
• Experience working with large distributed datasets and building reusable analytical dashboards or metrics layers (preferred).
• Advanced proficiency with big-data technologies and cloud platforms like Azure Data Explorer, Databricks, or Power BI (preferred).
• 6+ years of experience in data science, machine learning, or product analytics (preferred).
• 4+ years of experience using SQL and Python or R to implement statistical or machine learning methods (preferred).