Agent Evaluation and Self-Improvement Engineer

What you'll do

• Design and set evaluation strategies for individual skills, MCP servers, and full autonomous agents.
• Build automated evaluation systems to detect regressions across harnesses before they reach users.
• Mine production traces to identify failures, latency hotspots, and new usage patterns for evaluation cases.
• Develop a self-improvement loop where agents propose and test changes to their own skills and configurations.
• Create realistic sandboxed replicas of enterprise environments like code repositories and internal APIs.
• Measure multi-agent swarm performance, including work distribution, handoff success, and permission integrity.
• Calibrate model and agent judges against human labels to prevent cheating or reward hacking.
• Shape open evaluation standards and benchmarks for the broader AI industry.

What we're looking for

• BS or MS in Computer Science, Engineering, or a related field, or equivalent experience.
• 12+ years of experience building large-scale systems, including ML or agent evaluation systems and data pipelines.
• Hands-on experience building and evaluating agents on an open or closed agent harness.
• Solid grounding in experiment design, including statistical significance, variance across repeated runs, and A/B testing.
• Experience setting technical direction across teams as a senior individual contributor.
• Proficiency in Python, Go, Rust, or similar languages.
• Experience building self-improving or self-evolving agent systems that improve from their own evaluation results (preferred).
• Experience shipping an agent benchmark, evaluation framework, or simulated test environment adopted by other teams (preferred).