ML Engineer, Automated Evaluation and Adversarial Design

What you'll do

• Translate qualitative quality notions into measurable, reproducible automated evaluation systems for single-turn and multi-turn agentic experiences.
• Build adversarial test suites targeting model failure modes like context loss, instruction forgetting, and cascading errors.
• Develop and execute stress test protocols to validate performance under atypical conditions and complex tool-use sequences.
• Ensure alignment between automated and human evaluation methods by identifying and resolving systematic disagreements.
• Scale the generation of adversarial test cases and stress tests using programmatic automation for multi-turn scenarios.
• Communicate evaluation findings and model readiness assessments to cross-functional partners to influence product decisions.

What we're looking for

• Bachelor's degree in Computer Science, Machine Learning, Statistics, or a related field.
• 4+ years of experience building or extending ML evaluation systems and quality assessment frameworks.
• Experience independently defining evaluation architecture for AI systems where the unit of analysis is a conversation or session.
• Experience designing adversarial or red-teaming test methodologies for ML models, specifically targeting multi-turn interactions.
• Experience with Python and ML frameworks like PyTorch or TensorFlow in production or near-production settings.
• Track record of owning technical direction for evaluation efforts across multiple features or product areas.
• Graduate degree in a relevant field (preferred).
• Experience evaluating user-facing AI features, productivity software, or agentic workflows (preferred).