ML Engineer - Automated Evaluation and Adversarial Design

What you'll do

• Define and own automated evaluation approaches for AI features, ensuring measurable assessments.
• Build adversarial test suites targeting model failure modes across single-turn and multi-turn interactions.
• Develop stress test protocols to validate performance under atypical input conditions, including extended conversations.
• Ensure alignment between automated and human evaluation methods by resolving systematic disagreements.
• Scale adversarial test case generation and execute stress tests using automation for efficiency.
• Influence model and feature quality decisions by communicating evaluation findings to cross-functional teams.

What we're looking for

• Bachelor’s degree in Computer Science, Machine Learning, Statistics, or related field.
• 4+ years of experience building ML evaluation systems and designing evaluation benchmarks.
• Experience defining evaluation architecture for AI systems with conversation-level analysis units.
• Expertise in designing adversarial test methodologies targeting multi-turn interaction failures.
• Proficiency with Python and ML frameworks (PyTorch, TensorFlow) in production settings.