| Microsoft Careers

What you'll do

• Partner with researchers to translate cutting-edge evaluation research into production systems.
• Deprecate saturated benchmarks and design procedurally-generated code evaluations for models.
• Build GitHub's community benchmark submission program and publish transparency reports.
• Design multi-tier evaluation frameworks from automated suites to expert human evaluation.
• Convert user feedback into candidate regression tests through systematic pipelines.
• Establish evaluation as a first-class discipline with rituals, dashboards, and cadences.

What we're looking for

• 5+ years of experience in technical program management or related field.
• 2+ years managing programs in machine learning and AI/ML evaluation.
• Deep firsthand experience with AI/ML evaluation methodologies including benchmark design, human evaluation frameworks, and automated scoring systems.
• Personal experience using AI coding tools like Copilot daily, understanding developer perspectives on tool quality.
• Understanding of software engineering workflows at scale and how AI integrates into these processes.
• Experience managing community or open-source programs in a technical context.
• Proven ability to navigate competing priorities and build commitment to common goals in ambiguous environments.