Agentic AI, Senior Software Development Engineer

What you'll do

• Design, build, and scale evaluation frameworks for agentic AI experiences.
• Build tracing, observability, and quality measurement systems for production AI agents.
• Create platform infrastructure that enables domain teams to build on the Evals platform.
• Integrate new AI evaluation capabilities into production systems with applied scientists and ML engineers.
• Manage scaling, reliability, performance optimization, incident response, and cost management for the evaluation layer.
• Translate emerging agentic AI paradigms and LLM tooling into practical platform innovations.
• Evolve methods for measuring AI quality, reliability, trustworthiness, and product impact.

What we're looking for

• 4+ years of backend engineering experience designing, shipping, and operating scalable production ML services.
• Experience building platform infrastructure or developer-facing services consumed by multiple teams.
• Hands-on experience with ML Evals and Observability frameworks (Databricks MLflow, LangSmith, Braintrust, Promptfoo, or similar preferred).
• Experience evaluating third-party AI platforms and making build vs. buy decisions for platform infrastructure.
• Fluency working across applied science, ML, product, design, and engineering disciplines.
• Comfort with LLMs, agentic systems, and evaluation frameworks, including orchestration tools like LangChain, LangGraph, or MCP.
• Advanced degree in Computer Science, Machine Learning, or a related field, or equivalent practical experience (preferred).
• Experience designing and operating evaluation pipelines including offline batch evals, online scoring, and cost/latency/signal tradeoffs (preferred).