Senior Agent Evaluation Engineer, AI Evaluation Platform

What you'll do

• Build and ship production features including APIs, SDKs, orchestration components, and evaluation runners.
• Develop and evaluate AI agents featuring multi-turn trajectories, tool-use chains, and non-deterministic outputs.
• Write evaluation suites and validate LLM judges against human labels to measure reliability.
• Productionize ML research by translating prototype code into reliable, scalable Python services.
• Create SDKs and abstractions to improve the developer experience for other teams evaluating models.
• Maintain operational ownership by writing tests, setting up CI, and monitoring production metrics.
• Identify and fix codebase issues such as flaky tests, slow builds, and outdated runbooks.

What we're looking for

• 4-8 years of software engineering experience shipping production software.
• Experience building LLM agents with tool use, multi-step workflows, context management, and error recovery.
• Experience designing agent evaluations including non-deterministic outputs, multi-trial measurement, and trajectory scoring.
• Ability to build LLM judges and validate them against human agreement.
• Proficiency in Python with FastAPI and Pydantic.
• Fluency with AI coding tools like Claude Code or equivalents.
• Experience working on developer platforms, internal tools, or SDKs (preferred).
• Production experience with LLM/agent systems, job orchestration frameworks, or distributed compute (preferred).