Senior Machine Learning Engineer, ML Systems Evaluation Engineering

What you'll do

• Use Large Language Models (LLMs) as judges to automatically evaluate model changes and conversation naturalness.
• Harness generative AI to create adversarial scenarios that identify edge cases for robust evaluation.
• Curate and evolve high-quality evaluation datasets for state-of-the-art conversational models.
• Manage large-scale datasets specifically designed for machine learning training and evaluation.
• Define and measure evaluation coverage for LLMs and agentic systems.
• Provide offline evaluation insights to drive model development and improve end-user experience.
• Develop scalable data solutions for natural language products while maintaining strict privacy standards.

What we're looking for

• 7+ years of professional experience applying machine learning to real-world problems and crafting scalable data solutions for natural language products.
• Proven experience managing large-scale datasets for ML training and evaluation.
• Excellent programming skills in Python.
• MS/PhD in Machine Learning, Computer Science, or equivalent experience in a related field.
• Deep domain knowledge in Conversational AI and understanding of the end-to-end ML product lifecycle (preferred).
• Expertise in defining and measuring evaluation coverage for Large Language Models and agentic systems (preferred).
• Track record of delivering large-scale, cross-functional ML product or platform outcomes (preferred).
• Experience with systems engineering and understanding of ML and software component interdependencies (preferred).