Senior AI and HPC Observability Engineer

Nvidia

Confirmed live yesterday High trust

Quick summary

Work type
On-site
Location
Santa Clara, CASeattle, WA
Salary
$152,000–$241,500 / yr
Employment
Full-time
Posted
2 days ago
Freshness
Confirmed live yesterday

Market check

Salary context

Competitive pay

How this pay compares to similar roles

Similar $198k
This role $197k
$141k $252k
below market most similar roles pay here above market

This role pays less than 52% of similar roles. Most pay $162,000–$233,850 — the blue band above. At the midpoint, this role pays about $197k versus about $198k for comparable roles.

Based on 240 similar postings.

Employer

About Nvidia

Nvidia is a leading designer of graphics processing units (GPUs) and system-on-chip units, powering gaming, professional visualization, data centers, and artificial intelligence workloads. Industry: Semiconductors & AI Computing

Nvidia currently has 1668 open roles on FindRole.

Listed pay typically runs $184,000–$287,500 across 657 roles with salary data.

Most-posted roles

View all roles at Nvidia

At a glance

TL;DR · Senior AI and HPC Observability Engineer

The Senior AI and HPC Observability Engineer joins the Managed AI Superclusters team to build and scale next-generation observability and telemetry platforms. You will design and develop high-throughput telemetry pipelines, high-performance backend services for ingestion and routing, and extend OpenTelemetry collectors, processors, and instrumentation libraries. Your daily work involves building metrics pipelines using large-scale time-series storage systems, optimizing platform reliability through capacity planning, and developing monitoring and alerting frameworks. You will utilize Python, Go, or Java to manage distributed systems and cloud-native infrastructure including Kubernetes. Key technical requirements include experience with PromQL, Kafka, Spark, or Flink to handle high-volume metrics, logs, and traces. This role specifically addresses the technical challenge of monitoring GPU workloads and AI/ML pipelines within high-performance computing environments.

What you'll do

  • Design and scale observability platforms for high-volume metrics, logs, and traces across distributed environments.
  • Build high-performance backend services for telemetry ingestion, processing, and routing.
  • Develop and extend OpenTelemetry collectors, processors, exporters, and instrumentation libraries.
  • Build and optimize metrics pipelines using large-scale time-series storage systems.
  • Design and operate real-time and batch telemetry pipelines using streaming and distributed data technologies.
  • Improve platform reliability and cost efficiency through tuning, capacity planning, and system optimization.
  • Develop monitoring, alerting, and service reliability frameworks to ensure platform health.

What we're looking for

  • Bachelor’s degree in Computer Science, Computer Engineering, or related field or equivalent experience.
  • 5+ years of experience building backend or distributed systems in production environments.
  • Strong programming skills in Python, Go, or Java for developing production-quality software.
  • Hands-on experience with modern observability architectures, including metrics, logs, and traces.
  • Solid experience with PromQL and time-series data systems.
  • Experience building or operating distributed data pipelines using technologies such as Kafka, Spark, or Flink.
  • Experience working with Kubernetes and cloud-native infrastructure.
  • Proven experience designing and scaling observability platforms for AI, GPU, or HPC environments (preferred).

More like this

Similar roles

Senior Software Engineer, AIOps and Observability

Nvidia

Santa Clara, CA 5 days ago $200,000–$322,000
Observability AIOps Prometheus Victoria Metrics Vector Loki Grafana Alert Manager Clickhouse OpenTelemetry BigPanda PagerDuty Datadog Kubernetes Nomad Docker Microservices NATS Kafka Go Python Java C# Machine Learning Generative AI LLMs Data Pipelines Cloud-native BareMetal
10+ yrs exp

Senior HPC AI Cluster Engineer

Nvidia

Remote (Santa Clara, CA) 73 days ago $176,000–$276,000
HPC AI Python Bash Kubernetes Slurm Ansible Jenkins CUDA InfiniBand RDMA Lustre GPFS Weka.io Linux Windows AWS Azure Google Cloud VMware KVM Ethernet
8+ yrs exp Remote

Senior Software Engineer, Agentic AI and Observability

Nvidia

Santa Clara, CA 54 days ago
Python Kubernetes Docker CI/CD Datadog OpenTelemetry Grafana Prometheus Terraform Pulumi LangChain LlamaIndex Semantic Kernel SRE DevOps GitOps Distributed Systems LLM Orchestration
8+ yrs exp

Senior AI Observability Engineer

Lam Research

Fremont, CA 65 days ago $92,000–$211,000
LLM LangGraph Semantic Kernel AutoGen Model Context Protocol MLOps LLMOps OpenTelemetry Vector Databases AWS Azure Cloud Platform CI/CD TCP/IP BGP OSPF VXLAN EVPN MPLS SD-WAN ServiceNow
8+ yrs exp Hybrid

Senior Application Engineer, HPC and AI for Physics

Nvidia

CA +1 68 days ago $140,000–$224,250
HPC AI Machine Learning CUDA GPU-accelerated computing Parallel Programming Numerical Simulations Data Processing Agentic AI Particle-in-cell ML Frameworks
5+ yrs exp Hybrid

Senior HPC Performance Engineer

Nvidia

Remote (OR) +3 18 days ago
CUDA Fortran C C++ OpenACC OpenMP MPI HPC Performance Analysis Assembly Linear Algebra Numerical Methods Compilers
8+ yrs exp Remote