AI Evaluations Engineer, US Decision Intelligence

What you'll do

• Architect a comprehensive Evals framework to trace agent responses, skill levels, and tool calling.
• Build and operate workflows to measure LLM output quality for chat, summarization, and recommendations.
• Implement rubric-based evaluations to score outputs for correctness, relevance, grounding, and consistency.
• Develop agent-as-a-judge patterns and harness-as-a-judge patterns applied against real traces in a sandbox.
• Instrument LLM and agent workflows to capture traces, prompts, metadata, and user feedback.
• Own the platform-wide eval gate by defining pass/fail contracts and holding releases to standards.
• Define agent-specific evaluations for task completion, tool correctness, and error recovery.
• Define rerun policies and variance thresholds for AI model performance.

What we're looking for

• 5+ years of experience in data and AI-related fields such as AI engineering, software development, ML engineering, data science, or QA roles.
• B.S. degree in Computer Science/Engineering, or equivalent work experience.
• Strong Python skills.
• Hands-on experience with AI evaluation techniques, such as Golden datasets, LLM-as-a-Judge, or rubric-based scoring.
• Experience with LLM ecosystems, RAG pipelines, and vector databases like Pinecone, FAISS, Milvus, or PostgreSQL.
• Proficiency in SQL and experience with at least one major data analytics platform such as Hadoop, Spark, or Snowflake.
• Experience with CI/CD or release validation workflows.
• Hands-on experience with Langfuse or similar tools for LLM observability.
• Sound communication skills and ability to message technical content to stakeholders (preferred).
• Familiarity with embeddings, retrieval algorithms, agents, and data modeling for vector and graph databases (preferred).
• Knowledge of distributed systems, asynchronous messaging, and caching technologies like RabbitMQ, Redis, and Valkey (preferred).
• Applied knowledge of GenAI, RAG strategies, microservices, recommendation systems, and context engineering (preferred).
• Working knowledge of agent evaluation concepts like trajectory vs. end-to-end vs. component-level evaluation (preferred).
• Advanced degree (MS or Ph.D.) in Economics, Electrical Engineering, Statistics, Data Science, or a similar quantitative field (preferred).