Senior Applied Scientist, AI Evaluation & Quality Systems

What you'll do

• Design and implement scalable ground truth generation pipelines across various task types and annotation modalities.
• Build and maintain real-time monitoring systems to detect drift and quality degradation in live pipelines.
• Design calibration frameworks to re-anchor LLM evaluators against human-verified gold sets.
• Build root-cause analysis tooling to identify disagreement patterns between automated and human judgments.
• Refine annotator training and guidelines based on findings from automated and human judgment analysis.
• Prototype, validate, and ship production-grade evaluation and quality-assurance systems.
• Incorporate feedback from ML teams and annotators into system design and architecture decisions.

What we're looking for

• 5+ years of industry experience in applied science or machine learning.
• Experience building or operating production-grade evaluation, annotation, or quality-assurance pipelines.
• Hands-on experience designing ground truth generation pipelines across varied task types and annotation modalities.
• Experience building real-time monitoring or anomaly/drift detection systems for live data or ML pipelines.
• Working knowledge of generative AI evaluation methodology, including LLM-as-a-judge, meta-evaluation, and calibration techniques.
• Proficiency in Python and ML frameworks with production experience deploying and monitoring LLM-based pipelines and agents.
• MS or PhD in Computer Science, Machine Learning, Statistics, or a related quantitative field, or equivalent practical experience.
• Experience designing configurable and extensible systems for non-builders (preferred).