Applied AI Scientist, Image Autograder Systems, Evaluation

What you'll do

• Develop autograders that reliably score image output quality at scale.
• Refine feature requirements, grading rubrics, and gold annotation sets with partner teams.
• Develop, evaluate, and iterate on grading prompts to align autograder behavior with established rubrics.
• Apply advanced techniques like fine-tuning to close accuracy gaps when prompt tuning reaches its limits.
• Design insightful analysis to measure and explain autograder quality.
• Build scalable systems to accelerate the autograder training processes.
• Deploy and drive the adoption of autograders across various AI feature teams.

What we're looking for

• Master's or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related field.
• Deep understanding of visual-language models.
• Familiarity with image quality assessment including perceptual quality dimensions, evaluation metrics, and rubric design.
• Proficiency in Python to write well-structured, production-ready model code.
• Strong communication skills to explain autograder quality and drive adoption with partner teams.
• 1+ years of industry experience in building VLM-based products (preferred).
• Familiarity with evaluator-specific concepts like grading accuracy, human rater agreement, calibration, and rubric design (preferred).
• Expertise in prompt tuning, fine-tuning for VLMs, reading AI literature, and building agentic systems (preferred).