Machine Learning Engineer, Human Centered AI - Evaluations & Insights

What you'll do

• Architect and execute comprehensive evaluation suites for LLMs and multimodal models to identify edge cases and safety issues.
• Develop deterministic and LLM-assisted scoring frameworks to quantify human-perceived quality metrics like helpfulness and hallucination rates.
• Translate qualitative failure modes into quantifiable loss patterns, programmatic guardrails, and actionable data-mixture adjustments for model training.
• Design scalable MLOps pipelines and distributed inference systems to automate regression testing and high-throughput model evaluation.
• Apply advanced ML techniques like embedding-based clustering to systematically map error taxonomies and latent failure manifolds.
• Refine model behavior by leveraging evaluation telemetry to inform prompt engineering, RAG strategies, and model fine-tuning.
• Build automated evaluation pipelines using LLMs to assess outputs at scale with high correlation to human annotations.

What we're looking for

• 5+ years of relevant industry experience in ML Engineering or Applied Research.
• Bachelor’s or Master’s degree in Computer Science, Machine Learning, Artificial Intelligence, Cognitive Science, or a related technical field.
• Advanced proficiency in Python and modern deep learning ecosystems like PyTorch, JAX, and Hugging Face.
• Proven experience building scalable ML inference pipelines, model-evaluation workflows, and structured rating frameworks for large-scale AI systems.
• Hands-on experience developing, fine-tuning, or evaluating LLMs, multimodal models, and NLP systems.
• Deep familiarity with AI quality metrics, hallucination detection, model alignment (RLHF/DPO), and LLM-as-a-judge frameworks.
• Experience building internal tools or automated pipelines for ML workflows using platforms like MLflow or Weights & Biases.
• Knowledge of human factors, HCI, or cognitive science methodologies as applied to AI system design (preferred).