Program Manager, Human Evaluations (Recommendations)

What you'll do

• Lead end-to-end execution of human evaluation and data operations initiatives from scoping to final delivery.
• Translate recommendation strategies into specific evaluation designs, success metrics, and project timelines.
• Manage human annotation vendors including capacity planning, SLAs, quality performance, and escalations.
• Develop scalable guidelines and scoring rubrics to ensure consistent rater judgment for subjective content.
• Oversee rater onboarding, calibration, and quality assurance workflows to maintain high data standards.
• Drive operational efficiency by monitoring cost per task, throughput, and turnaround times.
• Pilot and refine evaluation tasks to improve inter-rater reliability and feedback quality.
• Act as a cross-functional lead to align product, ML, and data science teams on evaluation goals.

What we're looking for

• 4+ years of experience in human evaluations, data collection, labeling, or annotation operations in ML environments.
• Experience managing human annotation vendors, raters, or data labeling teams, including performance and cost management.
• Experience improving operational efficiency of annotation programs regarding throughput, turnaround time, and cost.
• Proven ability in end-to-end management of complex, cross-functional programs with strong Program Management skills.
• Strong understanding of evaluation design, including guidelines, rubrics, and scoring protocols.
• Working fluency in film and TV content, including genres, formats, and talent.
• Experience with human labeling platforms and excellent written and verbal communication skills.
• Experience with personalization, recommender systems, search, or ranking evaluation (preferred).