Senior Machine Learning Engineer, Applied Scientist

What you'll do

• Lead applied science strategy for AI products from opportunity identification through launch and continuous improvement.
• Define success metrics for AI experiences including customer value, reliability, safety, and business impact.
• Design and operationalize LLM evaluation frameworks across the entire product development lifecycle.
• Build evaluation methodologies for agentic, retrieval-augmented, and multimodal AI systems.
• Develop offline experimentation pipelines to compare models, prompts, retrieval strategies, and system behaviors.
• Partner with engineering and product leadership to align science priorities with roadmap decisions.
• Collaborate with software engineering teams to build scalable, reliable, and maintainable production AI systems.
• Establish quality bars for AI launches using evaluation scorecards, regression testing, and guardrail metrics.

What we're looking for

• Bachelor's Degree in a related field and 6+ years of related experience.
• Master's Degree in a related field and 4+ years of related experience.
• Doctorate in a related field and 3+ years of related experience.
• Ability to pass the Microsoft Cloud background check.
• Significant experience building, evaluating, and shipping AI/ML-powered products or services (preferred).
• Hands-on experience with large language models, including prompt engineering, RAG, agents, and evaluation-driven iteration (preferred).
• Deep understanding of LLM evaluation throughout the development lifecycle, including dataset creation and benchmark design (preferred).
• Solid software engineering skills and experience collaborating with engineers to build production-quality AI systems (preferred).