Machine Learning Engineering Manager, Evaluation, Responsible AI and Safety

What you'll do

• Develop metrics to evaluate safety and fairness risks in generative models and features.
• Design datasets and identify data needs using human and synthetic generation methods.
• Develop sampling strategies combining human annotation with auto-grading for high-confidence insights.
• Implement data pipelines to execute end-to-end safety evaluations for multimodal models.
• Automate manual workflows and scale infrastructure to optimize evaluation velocity for product releases.
• Distill project findings into actionable insights for product engineering and safety policy teams.
• Analyze evaluation results and conduct failure investigations to improve feature safety.
• Present safety metrics and experimental interpretations to senior leadership and stakeholders.

What we're looking for

• MS or PhD in a relevant science or technology field with 4+ years of experience, or a BA/BS with 8+ years of experience.
• At least 2 years of experience directly managing engineering teams.
• Experience gathering and analyzing multi-modal data, including human- and LLM-generated data.
• Experience with LLM-Judges and human-in-the-loop techniques, including designing annotation projects and writing guidelines.
• Proficiency in data science, machine learning, analytics, Python, Pandas, and AI-assisted coding tools.
• Cross-functional collaboration skills to prioritize projects, maintain schedules, and communicate with all levels of stakeholders.
• A rigorous approach to metric development, experimental interpretation, and executive presentations.
• Experience in the Responsible AI space (preferred).