Machine Learning Engineer, Evaluation - Responsible AI and Safety

What you'll do

• Develop metrics to evaluate safety and fairness risks in generative models and features.
• Design datasets and identify data needs using human and synthetic generation methods.
• Develop sampling strategies combining human annotation with auto-grading for high-confidence insights.
• Implement data pipelines to execute end-to-end safety evaluations for multimodal models.
• Automate manual workflows and scale infrastructure to optimize evaluation velocity for product releases.
• Distill project findings into actionable insights for product engineering and safety policy teams.
• Conduct failure investigations and analyze evaluation results to improve feature safety.
• Present safety metrics and experimental interpretations to senior leadership and stakeholders.

What we're looking for

• MS or PhD in a relevant science or technology field with 4+ years of experience, or a BA/BS with 8+ years of experience.
• At least 2 years of experience directly managing engineering teams.
• Experience gathering and analyzing multi-modal data, including human- and LLM-generated data.
• Experience with LLM-Judges and human-in-the-loop techniques, including designing annotation projects and writing guidelines.
• Proficiency in data science, machine learning, analytics, and programming with Python and Pandas.
• Cross-functional collaboration skills to prioritize projects, maintain schedules, and communicate with stakeholders.
• Experience working in the Responsible AI space (preferred).
• Experience with multimodal models, agentic systems, or compliance and regulatory needs in the AI space (preferred).