Senior Machine Learning Engineering Manager, Evaluation

What you'll do

• Architect and build scalable evaluation systems for foundation models, including benchmarks, LLM-based evaluators, and simulation environments.
• Establish an end-to-end evaluation flywheel that connects observed model failures to targeted refinements and post-training improvements.
• Lead and mentor a team of machine learning engineers while remaining hands-on with technical design, experimentation, and code reviews.
• Define the technical strategy for automatic optimization of prompts, contexts, tools, rubrics, and agent-harnesses.
• Develop methods to convert evaluation findings into actionable signals like targeted datasets, synthetic trajectories, and reward signals.
• Design and scale synthetic data generation pipelines for both evaluation and post-training purposes.
• Adapt recent research in LLM-as-judge, reward modeling, and test-time search into production-quality workflows.

What we're looking for

• 8+ years of professional experience in machine learning, applied research, or software engineering.
• 3+ years of technical leadership experience, including direct people management of machine learning or software engineers.
• Master’s or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related technical field.
• Strong hands-on programming skills in Python and experience building reliable ML pipelines using modern deep learning frameworks.
• Deep experience with large language models or agentic systems, including evaluation of multi-turn behavior, tool use, and reasoning.
• Experience building automated evaluation methods such as LLM-based judges, rubrics, reward models, or scalable benchmark infrastructure.
• Experience with at least one model or agent refinement area such as automatic prompt optimization, post-training, or reinforcement learning.
• Track record of applying recent machine learning research to production systems or high-impact product development (preferred).