ML Engineer - Evaluation Analysis, Metric and Data Strategy

What you'll do

• Define and own quality metrics frameworks for AI features and agentic experiences.
• Analyze evaluation outputs to identify trends, regressions, and segment-level patterns.
• Drive data collection strategies with partner teams to ensure real-world relevance.
• Audit evaluation data representativeness to reflect actual user distributions accurately.
• Deliver concise metric summaries for leadership to inform decision-making processes.
• Influence model development direction by providing actionable feedback on failure patterns.
• Design multi-turn evaluation frameworks and session-level scoring rubrics.

What we're looking for

• Bachelor’s degree in Statistics, Data Science, Applied Mathematics, Computer Science, or related quantitative field.
• 5+ years of experience defining and operationalizing quality metrics in applied science, data science, or evaluation research.
• Experience with statistical analysis methods including significance testing, sampling design, effect size estimation, and experimental design.
• Ability to work with production user data, understanding biases and limitations compared to controlled evaluation data.
• Track record of independently designing metrics frameworks and driving data-informed decisions across cross-functional teams.
• Proficiency in Python (pandas, scipy, scikit-learn) or R for data analysis and visualization.