Software Engineer L5, AI Observability & Agent Evaluation

What you'll do

• Build an observability framework to provide metrics, logs, and distributed traces for ML and GenAI systems.
• Develop primitives for teams to monitor model performance, data quality, drift, and degradation independently.
• Create evaluation frameworks for LLM and agentic systems to measure response quality, grounding, and task success.
• Lead build-vs-buy evaluations and manage SDKs and APIs to integrate third-party observability and evaluation tools.
• Develop reusable libraries, SDKs, and templates to establish observability-by-default for new AI systems.
• Provide dashboarding, alerting, and SLO/SLI building blocks to track model latency, cost, and reliability.

What we're looking for

• Experience in software, AI/ML, or platform engineering with hands-on production observability, monitoring, or ML/LLM evaluation.
• Proven track record of designing standards, libraries, SDKs, or frameworks that other teams build on and adopt.
• Strong coding skills in Python and at least one of Java, Go, or Scala to build production services.
• Practical experience operating ML models in production, including drift, model performance, and data quality.
• Hands-on experience with modern observability stacks such as Prometheus, Grafana, Datadog, OpenTelemetry, ELK, or Jaeger.
• Solid understanding of distributed systems, microservices, and at least one major cloud platform like AWS, GCP, or Azure.
• Experience with vendor integration and VPC deployment.
• Hands-on experience with ML/LLM observability tools or building and evaluating LLM/GenAI and agentic systems (preferred).