Senior Software Engineer, AI, Evaluation

What you'll do

• Own the end-to-end architecture, roadmap, and reliability of core evaluation systems for multiple features.
• Design testing and validation strategies to identify system failures before they reach internal partners.
• Build and maintain integrations with annotation, compute, reporting, and benchmarking platforms.
• Convert recurring needs from teams building LLM judges into scalable product features.
• Establish standards and best practices for using AI coding agents while maintaining high code quality.
• Interpret evaluation results including human agreement, consistency, and data privacy constraints.
• Translate ambiguous requirements into focused technical plans to support both research and production cycles.

What we're looking for

• BS/MS/PhD in Computer Science, Machine Learning, or a related field.
• 8+ years of software engineering experience, including owning production systems used by other teams.
• Exceptional Python skills and strong understanding of system design, API design, testing, debugging, and monitoring.
• Experience shipping production code with AI coding agents while maintaining high quality.
• Proficient in building agentic systems, LLM-based applications, LLM-as-judge systems, and offline evaluations (preferred).
• Ability to interpret evaluation results including human agreement, consistency, and data privacy constraints (preferred).
• Expert in building and maintaining technical integrations across multiple platforms in a large organization (preferred).
• Product-minded ability to turn ambiguous requirements into a focused plan (preferred).