Principal AI Software Engineer

Palo Alto Networks

Confirmed live yesterday High trust

Quick summary

Work type
On-site
Location
Santa Clara, CA
Salary
$223,000–$306,500 / yr
Posted
2 days ago
Freshness
Confirmed live yesterday

Market check

Salary context

Above market

How this pay compares to similar roles

Similar $203k
This role $265k
$123k most similar roles pay here $326k

This role pays more than 85% of similar roles. Most pay $174,600–$231,000 — the shaded band above. At the midpoint, this role pays about $265k versus about $203k for comparable roles.

Based on 240 similar postings.

Employer

About Palo Alto Networks

Palo Alto Networks is a global cybersecurity company offering network security, cloud security, and endpoint protection solutions.

Palo Alto Networks currently has 42 open roles on FindRole.

Listed pay typically runs $164,000–$266,000 across 40 roles with salary data.

Most-posted roles

View all roles at Palo Alto Networks

At a glance

TL;DR · Principal AI Software Engineer

Principal AI Software Engineer joins the Information Technology team to lead a Generative AI transformation by architecting next-generation business applications and intelligent, agentic workflows. This technical leadership role involves designing a proxy-first AI ecosystem with unified API endpoints to ensure vendor independence across various LLMs like GPT-4, Llama 3, and Anthropic. The engineer will build observability pipelines for trace-level data, implement multi-agent systems, and develop automated evaluation frameworks using "LLM as a Judge" methodologies. Key responsibilities include automating Go-To-Market processes, establishing safety guardrails against prompt injections, and optimizing costs through smart model routing. Required skills include expert-level Python proficiency, experience with orchestration frameworks, and deep knowledge of multi-agent systems. The role addresses the challenge of integrating advanced AI into internal and external business logic while ensuring security and performance in a high-scale environment.

What does a AI Software Engineer earn?

Median $196000 from 30 postings across 13 companies.

See salary data

What you'll do

  • Architect a "Proxy-First" AI Gateway to ensure vendor independence and seamless switching between different LLM providers.
  • Build unified API endpoints that abstract the complexities of multiple LLM providers into a single interface.
  • Implement comprehensive observability pipelines to track trace-level data, prompt inputs/outputs, and model latency.
  • Design sophisticated Multi-Agent Systems and Agent-to-Agent (A2A) workflows to automate complex business processes.
  • Develop "LLM as a Judge" testing methodologies to automate quality assurance for system outputs.
  • Build robust guardrails to filter content and prevent PII exposure or prompt injection attacks.
  • Create automated detection mechanisms using behavioral analysis and adversarial training to protect model instructions.

What we're looking for

  • Bachelor's degree or equivalent experience in Computer Science or a related field.
  • 12+ years of IT system architecture, design, and development experience, or 8+ years with a Master's degree, or 5+ years with a PhD.
  • Proven track record of technical leadership in Agile/Scrum environments delivering large-scale implementations.
  • Expert-level proficiency in Python.
  • Extensive, hands-on experience building applications using Large Language Models (LLMs) such as Llama 3, GPT-4, and Anthropic.
  • Deep experience with Multi-Agent systems, Agent-to-Agent (A2A) communication, and orchestration frameworks.
  • Experience partnering with business stakeholders to design workflows for Go-To-Market, sales, or support operations (preferred).
  • Familiarity with Java or Node.js for enterprise integration (preferred).

More like this

Similar roles

Principal AI Software Engineer

Palo Alto Networks

Santa Clara, CA 113 days ago $223,000–$306,500
Generative AI LLMs Python Multi-Agent Systems GPT-4 Llama 3 Anthropic LangSmith Arize HoneyHive OpenTelemetry Java Node.js Agile Scrum API Design
10+ yrs exp

Principal AI Engineer

Wells Fargo

Boston, MA +2 10 days ago $191,000–$305,000
Generative AI LLMs Agentic AI RAG Python Java LangChain AutoGen Semantic Kernel Google Vertex AI OpenShift Docker Kubernetes Microservices CI/CD Natural Language Processing Computer Vision Deep Learning
7+ yrs exp Hybrid

Principal AI Development Engineer

Broadcom

San Jose, CA 164 days ago $118,800–$190,000
Generative AI RAG LangGraph CrewAI Python Fullstack Development Vector Databases Pinecone Weaviate Milvus Kubernetes Docker CI/CD GitHub Test-Driven Development (TDD)
10+ yrs exp

Principal AI Software Engineer

Amd

San Jose, CA 164 days ago $240,000–$360,000
ROCm CUDA OpenCL GPU Architecture CPU Architecture Diffusion MoE Compiler Operating Systems Performance Engineering Hardware/Software Co-design AI/ML Frameworks Open Source

Principal AI Software Engineer

Microsoft

15 days ago $142,800–$274,800
C C++ Python CUDA Linux Kernel Memory Management NVIDIA GPU NCCL GPUDirect Storage GPUDirect RDMA vLLM SGLang TensorRT-LLM CXL Distributed Systems Virtualization NUMA DMA I/O Stacks
6+ yrs exp Hybrid

Principal Software Engineer, Core AI Platform

JPMorgan Chase

Seattle, WA 32 days ago
Python AI Infrastructure Machine Learning Cloud Platforms Distributed Systems SRE Observability System Design SDKs Automated Testing Infrastructure Engineering Generative AI
10+ yrs exp