Director, Product Management – AI Inference Platform

Arm Holdings

Hybrid

Quick summary

Work type: Hybrid
Location: San Jose, CA
Salary: $260,000–$351,800 / yr
Posted: 2 days ago
Nearby: 99+ roles within 25 mi

Market check

Salary context

Above market

How this pay compares to similar roles

Similar $229k

This role $306k

$162k most similar roles pay here $372k

This role pays more than 89% of similar roles. Most pay $194,017–$264,225 — the shaded band above. At the midpoint, this role pays about $306k versus about $229k for comparable roles.

Based on 239 similar postings.

Employer

About Arm Holdings

Arm Holdings plc is a leading British semiconductor and software design firm, established in 1990 and recognized for developing energy-efficient processor architectures that power nearly all smartphones and a vast range of IoT and computing devices.

Arm Holdings currently has 38 open roles on FindRole.

Listed pay typically runs $184,500–$249,600 across 38 roles with salary data.

Most-posted roles

View all roles at Arm Holdings

At a glance

TL;DR · Director, Product Management – AI Inference Platform

Apply Now Log in to save

As a Director/Principal Product Manager on Arm’s Cloud AI team, you will lead the strategic development and execution of an advanced AI Inference Platform, focusing on how modern AI models are executed, served, and optimized across distributed environments. Your responsibilities include defining the product vision, driving innovation in performance and scalability, and partnering with engineering teams to deliver cutting-edge AI systems. You will manage the platform’s control plane services, translate technical concepts into customer value, and track key metrics for evolving AI workloads. This role requires expertise in AI/ML infrastructure, cloud platforms, and distributed systems, along with a deep understanding of large-scale serving architectures and LLM inference concepts. Familiarity with accelerator-based systems and modern inference frameworks is essential, as you will shape the future of AI infrastructure at Arm by addressing complex challenges related to performance, scalability, and usability.

Skills

AI/ML infrastructure cloud platforms distributed systems AI inference systems LLM inference concepts accelerator-based systems TensorRT-LLM vLLM production-scale AI systems CI/CD Kubernetes Docker AWS GCP Azure PostgreSQL MongoDB

What you'll do

Define and drive product vision, strategy, and roadmap for large-scale AI inference systems.
Own product direction across compute execution, inference serving, and control plane systems.
Drive inference architecture and orchestration strategy, including distributed serving and scheduling.
Define capabilities for KV cache management, memory optimization, and token lifecycle efficiency.
Shape development of inference platform capabilities delivering gains in latency, throughput, and cost efficiency.

What we're looking for

Proven experience in AI/ML infrastructure or distributed systems product management.
Deep understanding of AI inference systems and large-scale serving architectures.
Solid grasp of LLM inference concepts including KV cache, token streaming.
Demonstrated ability to deliver scalable, high-performance platform products.
Strong technical expertise with accelerator-based systems and performance optimization.

Similar roles

Careers

Qualcomm

San Diego, CA 145 days ago

NPU Machine Learning Product Management Engineering Research Software Development Technical Marketing Conferences Customer Feedback ROI Analysis Innovation Strategies IP Strategies Cross-functional Collaboration Program Management Milestones Budget Management P&L Influence Long-term Business Plans

Save

Director, Product Management- Enterprise AI

Capital One Financial

McLean, VA +3 54 days ago $209,500–$239,100

AI ML ProductManagement AgenticFrameworks LLMOrchestration CI/CD CloudServices Python R Java PostgreSQL Kubernetes Docker GitHub Jira Confluence Slack Miro Terraform AWS Azure GCP

Save

Sr. Director, Product Management - AI/ML Platform Intelligence

Capital One Financial

McLean, VA +3 106 days ago $245,100–$279,800

Kubernetes Observability Security AI/ML Experiment Management Model Monitoring Gen AI Application Evaluation MLOps AIOps Public Cloud Open Source Technologies CI/CD Python Java PostgreSQL AWS Azure GCP

Save

Director Product Management (AI and Automation)

Capital One Financial

McLean, VA +2 38 days ago $230,400–$263,000

AI LLMs RPA APIs Data Privacy Bias Mitigation Transparency Precision Recall Automation Touch-Rates CI/CD Python Java PostgreSQL Kubernetes AWS Azure Git JIRA Confluence Selenium Docker Terraform

Save

Director, Product Management - Agentic Product

Adobe

San Jose 1 day ago $231,000–$334,500

ABTesting GrowthModels DataAnalysis CreativeWorkflows PLGApproach CI/CD MCP PluginArchitecture UIUXDesign FreemiumModels VerticalMarketplace AgentAsAService ExecutiveReporting ProductMonetization CollaborationTools AdobeCloud

Save

Director, Product Management - Threat AI Detections

Cisco

Remote (San Jose, CA) 2 days ago $230,100–$325,300

AI/ML CI/CD Cloud Infrastructure Kubernetes Terraform Python PostgreSQL Docker Networking Protocols Traffic Analysis Security Architectures Cybersecurity Product Management Large-Scale Network Detection Services Advanced Threat Detection Platforms

Remote

Save