Principal Software Engineer, At-Scale Reliability and Fleet Intelligence

Nvidia

Confirmed live today High trust

Quick summary

Work type
On-site
Location
Santa Clara, CA
Salary
$272,000–$431,250 / yr
Employment
Full-time
Posted
1 day ago
Freshness
Confirmed live today

Market check

Salary context

Above market

How this pay compares to similar roles

Similar $203k
This role $352k
$108k $466k
below market most similar roles pay here above market

This role pays more than 99% of similar roles. Most pay $174,600–$231,000 — the blue band above. At the midpoint, this role pays about $352k versus about $203k for comparable roles.

Based on 240 similar postings.

Employer

About Nvidia

Nvidia is a leading designer of graphics processing units (GPUs) and system-on-chip units, powering gaming, professional visualization, data centers, and artificial intelligence workloads. Industry: Semiconductors & AI Computing

Nvidia currently has 1623 open roles on FindRole.

Listed pay typically runs $184,000–$287,500 across 1110 roles with salary data.

Most-posted roles

View all roles at Nvidia

At a glance

TL;DR · Principal Software Engineer, At-Scale Reliability and Fleet Intelligence

The Principal Software Engineer, At-Scale Reliability and Fleet Intelligence — CSP Engagements joins the CSP Engagements team as a technical focal point for fleet-scale reliability. This role involves working directly with engineering teams of key CSP and hyperscale customers to ensure platforms achieve target MTBI in production. You will drive work streams to build a shared understanding of reliability software and firmware architecture, incorporate fleet telemetry into improvement priorities, and validate lab results in customer environments. Key responsibilities include conducting statistical failure analysis using Pareto, survival analysis, and Weibull methods, and developing predictive failure models. Required skills include expertise in multi-NUMA, rack-scale system software, firmware, and time-series databases. You will solve systemic architectural gaps and hardware failure modes across compute, interconnect, memory, power, and thermal domains.

What does a Software Engineer earn in California?

Median $214500 from 1193 postings across 76 companies.

See salary data

What you'll do

  • Drive reliability work streams with CSP engineering teams to align on MTBI measurement, failure classification, and health monitoring architecture.
  • Synthesize CSP fleet reliability data to identify cross-customer failure patterns and champion improvements to firmware and hardware teams.
  • Define consistent MTBI measurement methodologies that function across diverse CSP monitoring environments and operational practices.
  • Conduct fleet-scale failure pattern analysis using statistical methods like Pareto and survival analysis to classify systemic issues.
  • Design fleet health monitoring integration architectures to align NVIDIA's telemetry and reporting with CSP automation workflows.
  • Define burn-in reliability test environments and cluster certification criteria in collaboration with quality teams.
  • Develop predictive failure models using fleet telemetry and validate their effectiveness in real customer environments.

What we're looking for

  • 15+ years of experience in systems software at datacenter scale or reliability engineering with a focus on at-scale challenges.
  • BS or MS in Computer Science, Electrical Engineering, Statistics, or a related field (or equivalent experience).
  • Deep expertise in multi-NUMA, rack-scale system software and firmware.
  • Proficiency in statistical failure analysis methods including MTBF/MTBI calculation, Pareto analysis, and root cause classification.
  • Experience with fleet-level telemetry and observability systems such as time-series databases, anomaly detection, and event correlation.
  • Understanding of hardware failure modes in large-scale GPU/accelerator deployments across compute, interconnect, memory, power, and thermal domains.
  • Experience defining or operating burn-in, stress testing, or certification frameworks for complex hardware systems.
  • Experience in fleet reliability at a hyperscaler (preferred).

More like this

Similar roles

Principal Software Engineer, Rack-Scale System Software

Nvidia

Remote (Santa Clara, CA) +3 4 days ago $272,000–$431,250
System Software Firmware Distributed Systems Fabric Management GPU NVSwitch NVOS Cluster Management Telemetry APIs Error Handling Fault Isolation API Design Driver Lifecycle Power Management InfiniBand NVLink
10+ yrs exp Remote

Principal Software Engineer, CSP Engagements

Nvidia

Santa Clara, CA 2 days ago $272,000–$431,250
System Software Architecture Linux Kernel Device Drivers x86 ARM aarch64 CUDA GPU Computing CXL Computer Architecture Performance Analysis Deep Learning High-Performance Computing Memory Fabric In-band Management
10+ yrs exp

Principal Software Engineer, GPU Firmware and GPU System Software

Nvidia

Remote (Santa Clara, CA) +3 4 days ago $272,000–$431,250
GPU Firmware GPU System Software VBIOS InfoROM Microcontroller Firmware GPU Driver NVLink Secure Boot Attestation Multi-tenancy Isolation Power Management Telemetry ECC Fleet Management A/B Updates Rollback Strategy
10+ yrs exp Remote

Principal Software Engineer, AI Infrastructure

Microsoft

6 days ago $142,800–$274,800
Distributed Systems AI/ML Serving Azure Python C++ Java C# JavaScript C Observability Telemetry Distributed Tracing Capacity Management Load Balancing Reliability Engineering High-Performance Computing
6+ yrs exp Hybrid

Senior Software Program Manager, CSP Engagements

Nvidia

Santa Clara, CA 4 days ago $168,000–$258,750
Systems Software Embedded Software Linux Kernel Drivers Firmware BMC BIOS UEFI Hardware/Software Integration platform bring‑up Observability High-Performance Computing rack scale systems
10+ yrs exp