Machine Learning Evaluation Engineer

What you'll do

• Design and build scalable evaluation pipelines for multi-modal models across text, image, and video tasks.
• Develop and calibrate subjective evaluation methods using LLM-as-Judge and human grading techniques.
• Perform deep failure analysis to identify root causes in models, prompts, and upstream software components.
• Create failure taxonomies and automated dashboards to streamline reporting and iterative experimentation.
• Curate high-quality evaluation datasets using auto-labeling, manual annotation, and synthetic data generation.
• Analyze dataset quality, diversity, and coverage gaps to ensure evaluation sets reflect real-world usage.
• Provide data-driven insights and metrics to model training teams to guide fine-tuning and iterations.

What we're looking for

• Bachelor's degree and a minimum of 3 years of relevant industry experience.
• 3+ years of applied experience in Machine Learning, Computer Vision, or AI System Evaluation.
• Deep understanding of core Machine Learning principles including probability, statistics, data distributions, and model bias/variance.
• Experience analyzing data quality, diversity, representativeness, and coverage gaps in large datasets to inform curation strategies.
• Applied experience evaluating generative models, including prompt tuning, subjective evaluation of open-ended outputs, and human-in-the-loop methodologies.
• Proven track record of defining robust metrics and designing evaluation frameworks for foundation models using LLM/VLM-as-a-Judge methodologies.
• Strong proficiency in Python and building automated pipelines for model endpoint integration, multi-step evaluation orchestration, and logging.
• Theoretical and practical understanding of Computer Vision and Vision-Language Models (preferred).