Evaluation Science Lead

What you'll do

• Define the long-term evaluation science strategy and roadmap for Globalization AI solutions across 50+ languages.
• Design robust statistical methodologies including power analysis, sampling, and significance thresholds to evaluate models.
• Architect scalable evaluation workflows that blend human annotation protocols with automated systems like LLM-as-judge.
• Expand evaluation criteria to incorporate behavioral and user engagement signals beyond core linguistic quality.
• Translate complex evaluation data into actionable recommendations and go/no-go evidence for Engineering and leadership.
• Build continuous performance monitoring and drift-detection systems to identify model regressions.
• Establish feedback loops with Quality Operations and Engineering to refine evaluation rubrics as models evolve.

What we're looking for

• 5+ years of experience in evaluation science, data science, or ML systems development owning evaluation systems at scale end-to-end.
• Experience applying statistical methodologies including sampling, significance testing, and confidence intervals.
• Practical understanding of measurement validity principles.
• Hands-on experience measuring annotator agreement, diagnosing divergence, and improving annotation protocols.
• Proficiency with statistical tools and languages including R, Python, and SQL.
• Experience designing evaluation frameworks adopted and scaled by operational teams.
• Strong written and verbal communication skills for technical and non-technical audiences.
• Master’s, PhD, or comparable experience in Statistics, Computational Linguistics, Computer Science, Psychometrics, Data Science, or related quantitative field (preferred).