Engineering Program Manager, Siri and Apple Intelligence Evaluation

You will work with top tier data scientists, engineers, research teams, and product teams across Apple to help ensure we deliver high-quality, safe, and beneficial AI-powered experiences that over 1 billion customers expect and love. This role requires technical depth in evaluation methodologies combined with strong program management expertise to drive comprehensive assessment of model capabilities, safety, helpfulness, and user experience quality. Lead the design and execution of evaluation programs for Siri and Apple Intelligence features, establishing comprehensive frameworks to assess model capabilities across multiple dimensions. Oversee the development and implementation of evaluation infrastructure that scales with rapidly evolving model capabilities and provides real-time insights during training and deployment cycles Drive consensus across multiple groups with varying opinions & needs. Analyze evaluation results to identify patterns, failure modes, and opportunities for model improvement, translating complex findings into actionable insights for product and engineering teams Foster a strong measurement-informed product development culture in SWE, influence partner organizations to develop rigor in measurement driven culture across Siri and Apple Intelligence team Coordinate evaluation efforts during critical modeling and product milestones, ensuring comprehensive coverage, timely results, and alignment with product launch schedules Provide clear, timely and objective communication on evaluation insights to executives across SWE and partner organizations to help inform critical product decisions and training strategies Establish best practices and standards for evaluation development across the organization, enabling rapid iteration on evaluation design and supporting both automated and human-in-the-loop assessment approaches Reflect on how the team operates, proactively propose and implement new ways to enable more effective processes & delivery of results that drive continuous improvement in evaluation quality Bachelor's degree in Statistics, Business Intelligence, Computer Science, other Quantitative Sciences, or related field and equivalent experience 8+ years of experience in driving large scale program building machine learning powered products or analytics to support product development 5+ years of experience managing programs in AI powered product space, preferably experience in evaluation of ML/AI products Ability to deal with ambiguities, drive disambiguation and clarities around evaluation methodologies, shepherd multiple teams to converge on rigorous measurement frameworks Experience designing and implementing evaluation systems for machine learning models, particularly large language models or conversational AI systems Program management skills including program structuring and managing multiple work streams interdependently across research, engineering, and product teams Problem-solving skills with attention to details in identifying edge cases, failure modes, and capability gaps Ability to communicate abstract ideas clearly, manage comprehensive yet succinct program status updates to all levels of audience, both verbally and in written forms Proven adaptability and agility in making adjustments to program strategy and plan with evolving model capabilities and product decisions Master's or PhD degree in Statistics, Machine Learning, Computer Science, other Quantitative Sciences, or related field and equivalent experience Experience with statistical analysis and drawing meaningful conclusions from large-scale evaluation datasets Deep understanding of LLM capabilities, limitations, and safety considerations Self-sufficient in analyzing and drawing conclusions about model quality, user experience, and product opportunity from raw and refined evaluation data Player-coach capable of personally leading large evaluation initiatives while coaching team members along the way and mentoring team members to grow evaluation expertise

For more details click Job Post.