below marketmost similar roles pay hereabove market
This role pays more than
98%
of similar roles. Most pay
$212,175–$288,187
— the blue band above.
At the midpoint, this role pays about
$404k
versus about
$250k
for comparable roles.
Based on 240 similar postings.
Employer
About Apple Inc
Apple Inc. is a multinational technology company known for designing and manufacturing consumer electronics, software, and online services, including the iPhone, Mac, iPad, and App Store. Industry: Consumer Electronics & Software
Apple Inc currently has
3497 open roles
on FindRole.
Listed pay typically runs
$174,149–$277,600
across 2702 roles with salary data.
JOB TITLE: Director, Evaluation and Quality Assurance, Services Special Projects
The Director, Evaluation and Quality Assurance, Services Special Projects will build and lead the evaluation and quality assurance function for a multi-year engineering initiative. As the first hire in this area, you will establish the strategy, tooling, and organizational structure from the ground up. You will define quality metrics for conversational understanding, relevance, ranking, and revenue-bearing correctness across the product surface. Responsibilities include designing automated grading systems, managing human evaluation programs, and integrating quality gates into release processes. You will utilize LLM-based autograders, rubrics, logging infrastructure, and A/B testing to monitor third-party API integrations and agentic systems. This role solves the problem of ensuring transaction integrity and conversational accuracy within a large, evolving real-world dataset for a product still taking shape.
Design metrics and automated grading systems to evaluate large, frequently changing datasets.
Integrate evaluation results into release processes as explicit quality gates for all product layers.
Prevent quality regressions across the product surface, from user-facing correctness to backend transaction integrity.
Define "quality" and "done" criteria in partnership with engineering and applied ML teams.
Build the evaluation infrastructure, tooling, and organizational structure from the ground up.
Manage the balance between automated grading systems and human evaluation programs.
Make strategic judgment calls on where to invest limited evaluation resources to reduce risk.
What we're looking for
12+ years of experience in data science, machine learning evaluation, quality assurance, or test engineering roles in a technical product environment.
15+ years of experience in data science, machine learning evaluation, quality assurance, or test engineering roles in a technical product environment (preferred).
Experience building and leading multi-level QA or evaluation organizations through managers.
Demonstrated experience building an evaluation or QA function from the ground up, including defining metrics, process, tooling, and organizational structure.
Extensive experience utilizing LLM-based autograders and rubrics, including rubric design, evaluation set development, validation, calibration, and human-model alignment.
Deep experience in traditional human evaluation methodology, logging, and A/B testing for consumer-facing products at scale.
Experience evaluating conversational AI, recommendation, ranking, or search systems, and ideally agentic or tool-use systems.
BS in Computer Science, Engineering, or a related technical field, or equivalent experience.
AWS
Azure
GCP
Microservices
Event-driven systems
AI
ML
Cloud-native architectures
Subscription management
Identity management
Recommendation systems
GDPR
CCPA
Commerce platforms
Systems Engineering
Robotics
Motion Control
Automation Software
Control Architecture
Atmospheric Systems
EFEM
Design Qualification
Verification and Validation
Reliability Testing
Root-Cause Analysis
Electrical Design