Evaluation and ML Systems Engineer, AI Safety and Security Engineering

What you'll do

• Build benchmarking and reproducibility systems to measure program progress and performance.
• Design metrics, baselines, and protocols to ensure fair comparisons of ML models.
• Develop infrastructure that maps every result directly to the specific code and runs that produced it.
• Automate repetitive measurement tasks to streamline data collection for the engineering team.
• Partner with security researchers and platform engineers to design trustworthy, reproducible experiments.
• Establish evidence-based systems to validate claims and ensure findings are reviewable before use.
• Develop Python-based infrastructure for experiment tracking and automated data pipelines.

What we're looking for

• Bachelor's degree or equivalent experience in a relevant field.
• 5+ years of experience in ML engineering or evaluation.
• Experience designing benchmarks, metrics, and statistically sound comparisons for ML systems.
• Demonstrated measurement rigor with a skeptical approach to metrics, baselines, and claims.
• Proficiency in Python engineering for shared infrastructure, experiment tracking, and data pipelines.
• Preferred experience evaluating security tooling or pipelines.
• Preferred experience measuring agentic systems or LLM behavior.
• Preferred history of contributing to public benchmarks or evaluation frameworks.