Agentic AI Evaluation Engineer

What you'll do

• Design and develop agent evaluation pipelines across different environments.
• Define and standardize metrics for assessing conversational AI quality.
• Build automated systems to evaluate agent performance continuously.
• Manage datasets, test sets, and annotation workflows for evaluations.
• Conduct experiments and A/B testing to improve agent quality.

What we're looking for

• 5-7 years of relevant work experience in AI or chatbot platforms
• Expertise in CI/CD pipelines and machine learning models
• Proficiency in designing evaluation metrics and benchmarks for AI agents
• Ability to build automated and human-in-the-loop evaluation systems
• Experience in managing datasets, test sets, and annotation workflows
• Understanding of responsible AI principles including bias and fairness mitigation