AI/ML Evaluation Specialist, Human Data

What you'll do

• Execute end-to-end human data collection programs for multilingual, multimodal, and multi-turn AI features.
• Design measurement frameworks and reporting systems to track spend, speed, inter-rater reliability, and volume.
• Build and implement quality frameworks using statistical process controls to detect and remediate data degradation.
• Analyze workflows to identify inefficiencies and implement automated pipelines or agent-in-the-loop solutions.
• Develop onboarding, calibration, and performance programs for internal and external workforces.
• Apply human-centered AI principles to reduce cognitive burden on annotators during task design.
• Ensure all data collection programs comply with legal, privacy, and security standards.
• Serve as the primary liaison between engineering, product, legal, and procurement teams to align on shared standards.

What we're looking for

• Bachelor's degree or higher in Cognitive Science, Linguistics, or a related field with an experimental or empirical component.
• 4+ years of experience leading cross-team human data programs for AI/ML in NLP/NLU or generative AI environments.
• Proficiency in programming and data languages including Python, R, and SQL to analyze large datasets and automate tasks.
• Hands-on experience managing 0→1 human-in-the-loop data collection, annotation, and evaluation initiatives using agentic workflows.
• Experience working with diverse data types such as speech, text, and multimodal content across multiple languages.
• Expertise in end-to-end data annotation quality management, including statistical process controls and quality metrics.
• Familiarity with privacy-preserving data handling practices and compliance frameworks.
• Proven ability to work cross-functionally with engineering, data science, legal, privacy, and third-party suppliers.