ML Engineer - Automated Evaluation and Adversarial Design

What you'll do

• Define and own automated evaluation approaches for AI features across single-turn and multi-turn interactions.
• Build adversarial test suites targeting known and emerging model failure modes, including edge cases in productivity workflows.
• Develop stress test protocols to validate performance under atypical conditions, such as extended conversation lengths and complex sequences.
• Ensure alignment between automated and human evaluation methods by identifying and resolving systematic disagreements.
• Scale adversarial test case generation and stress test execution using automation for multi-turn scenarios and agent interaction traces.

What we're looking for

• Bachelor’s degree in Computer Science, Machine Learning, Statistics, or related field
• 4+ years of experience building ML evaluation systems and designing evaluation benchmarks for sequential AI outputs
• Experience independently defining evaluation architecture and methodology for AI systems with multi-turn interaction focus
• Expertise in designing adversarial test methodologies targeting failures across multi-turn interactions
• Proficiency in Python and ML frameworks (PyTorch, TensorFlow) in production settings