Senior Software Engineer, Agentic Evaluation

What you'll do

• Develop infrastructure, tooling, and pipelines to evaluate Siri reliably at scale.
• Extend evaluation capabilities across multiple devices, platforms, and runtime environments.
• Translate ambiguous early requirements into concrete, automated coverage for new features and interaction modalities.
• Diagnose failures across the stack from environment provisioning through pipeline execution and scoring.
• Implement auto-diagnostics and drive durable fixes for system failures.
• Contribute to architectural decisions for the team's evaluation systems.
• Design and operate large-scale test infrastructure including device provisioning and continuous integration.
• Evaluate ML, LLM, and agent-based systems using specific metrics and scoring methodologies.

What we're looking for

• Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field (or equivalent experience).
• Strong programming skills in at least one compiled language such as Swift, C++, or Objective-C.
• Proficiency in Python for tooling and automation.
• Solid understanding of computer science fundamentals.
• Experience evaluating ML, LLM, or agent-based systems including metrics and scoring methodology.
• Experience designing or operating large-scale test infrastructure and continuous integration systems.
• Experience staging and provisioning test environments to produce repeatable, deterministic conditions.
• Ability to communicate effectively with both technical and program audiences while navigating ambiguous requirements.