Annotation Data Scientist, Evaluation Integrity (Siri)

What you'll do

• Design HITL annotation tasks to assess the quality of user agent personae and validity of conversations.
• Author and maintain rubrics for human grading aligned with agentic evaluators and product guidelines.
• Manage multiple annotation programs end-to-end from requirements gathering to stakeholder delivery.
• Develop custom annotation tooling in partnership with software engineers to support evaluation tasks.
• Apply data science techniques to analyze human-labeled data, measuring evaluator accuracy and reliability.
• Translate annotator feedback into improvements for user agents and automated evaluators.

What we're looking for

• Bachelor's or Master's degree in a quantitative field or equivalent experience.
• 5+ years of hands-on experience with human-in-the-loop evaluation methodologies for ML systems.
• Expertise in Python for data processing, analysis, and prototyping using libraries like pandas and Jupyter.
• Experience designing and implementing annotation schemas and rubrics for machine learning training or evaluation.
• Ability to manage multiple concurrent dataset curation efforts, including coordinating with annotators and monitoring performance metrics.