Senior Machine Learning Engineering Manager, Evaluation

What you'll do

• Architect and build scalable evaluation systems including benchmarks, LLM-based evaluators, and simulation environments.
• Establish an end-to-end evaluation flywheel that connects observed failures to targeted model and agent refinements.
• Lead and mentor a team of machine learning engineers while remaining involved in technical design and code reviews.
• Define the technical strategy for automatic optimization of prompts, context, tools, and rubrics.
• Convert evaluation findings into actionable signals such as reward models, preference signals, and synthetic trajectories.
• Design and scale synthetic data generation pipelines for both model evaluation and post-training.
• Translate recent research in LLM evaluation and agentic systems into production-quality workflows.

What we're looking for

• Master's or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related technical field.
• 8+ years of professional experience in machine learning, applied research, or software engineering.
• 3+ years of technical leadership experience including direct people management and mentoring of engineers.
• Strong hands-on programming skills in Python and building reliable ML pipelines with modern frameworks.
• Deep experience with large language models or agentic systems, including multi-turn behavior and tool use.
• Experience building automated evaluation methods such as LLM-based judges, reward models, or simulation-based evaluation.
• Experience with model refinement areas like automatic prompt optimization, post-training, or reinforcement learning.
• Proven ability to communicate complex technical trade-offs and align research, engineering, and product teams.