Machine Learning Engineer, Evaluation and Quality, Creativity Apps

What you'll do

• Define evaluation strategies and metrics for ML models, LLMs, and AI features tailored to specific product use cases.
• Curate and create high-quality evaluation datasets and manage iterative improvement loops to ensure data remains representative.
• Design, build, and optimize scalable evaluation infrastructure with reproducible runs and tracked baselines.
• Apply LLMs and agentic workflows to automate judging, test generation, and analysis for more efficient evaluation.
• Drive end-to-end quality from model checkpoints to shipping features, including triaging and resolving issues.
• Translate evaluation results into clear ship/no-ship risk recommendations for both technical and non-technical audiences.

What we're looking for

• 5+ years of industry experience in machine learning or software engineering, focusing on ML/AI model evaluation and quality.
• Experience with evaluation methodology for ML models, LLMs, and AI products or features.
• Strong understanding of the ML development cycle, including data, training, evaluation, deployment, and monitoring.
• Solid machine learning fundamentals and LLM knowledge, including agentic workflows.
• Track record of applying LLMs or agentic workflows to ML evaluation and quality.
• Experience in data curation, dataset creation, and dataset quality with iterative improvement workflows.
• Experience designing, implementing, maintaining, or optimizing ML evaluation infrastructure at scale.
• Bachelor's degree in Computer Science, Data Science, Machine Learning, or a related field, or equivalent experience.