ML Engineer - Automated Evaluation and Adversarial Design

What you'll do

• Define and own automated evaluation approaches for AI features, ensuring measurable assessments.
• Build adversarial test suites targeting model failure modes across multi-turn interactions.
• Develop stress test protocols to validate performance under demanding conditions.
• Align automated and human evaluation methods, resolving systematic disagreements.
• Scale adversarial test case generation and execute stress tests programmatically.
• Influence model quality decisions by communicating evaluation findings to partners.

What we're looking for

• Bachelor’s degree in Computer Science, Machine Learning, Statistics, or related field
• 4+ years of experience building ML evaluation systems and designing evaluation benchmarks
• Experience independently defining evaluation architecture for AI systems with multi-turn analysis
• Expertise in designing adversarial test methodologies targeting failures across multi-turn interactions
• Proficiency in Python and ML frameworks (PyTorch, TensorFlow) in production settings
• Track record of owning technical direction for evaluation efforts across multiple features