ML Engineer, Automated Evaluation and Adversarial Design

What you'll do

• Define and own automated evaluation approaches that translate qualitative quality into measurable metrics for multi-turn agentic experiences.
• Build adversarial test suites targeting model failure modes like context loss, instruction forgetting, and cascading errors.
• Develop and execute stress test protocols to validate performance under atypical conditions and complex tool-use sequences.
• Ensure alignment between automated and human evaluation methods by identifying and resolving systematic discrepancies.
• Scale the generation of adversarial test cases and stress tests using programmatic automation for multi-turn scenarios.
• Communicate evaluation findings and model readiness assessments to cross-functional partners to influence product decisions.

What we're looking for

• Bachelor's degree in Computer Science, Machine Learning, Statistics, or a related field.
• 4+ years of experience building or extending ML evaluation systems and designing quality assessment frameworks.
• Experience independently defining evaluation architecture for AI systems where the unit of analysis is a conversation or session.
• Experience designing adversarial or red-teaming test methodologies for ML models, specifically targeting multi-turn interaction failures.
• Experience with Python and ML frameworks like PyTorch or TensorFlow in production or near-production settings.
• Track record of owning technical direction for evaluation efforts across multiple features or product areas.
• Graduate degree in a relevant field (preferred).
• Experience evaluating user-facing AI features, productivity tools, or agentic workflows (preferred).