Staff Software Engineer, ML Observability

Datadog

Confirmed live 2 days ago High trust
Hybrid

Quick summary

Work type
Hybrid
Location
—
Salary
$234,000–$300,000 / yr
Posted
101 days ago
Freshness
Confirmed live 2 days ago

Market check

Salary context

Above market

How this pay compares to similar roles

Similar $228k
This role $267k
$162k most similar roles pay here $315k

This role pays more than 82% of similar roles. Most pay $199,587–$256,000 — the shaded band above. At the midpoint, this role pays about $267k versus about $228k for comparable roles.

Based on 240 similar postings.

Employer

About Datadog

Datadog, Inc. is an American company that provides an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform.

Datadog currently has 266 open roles on FindRole.

Listed pay typically runs $161,000–$205,000 across 138 roles with salary data.

Most-posted roles

View all roles at Datadog

At a glance

TL;DR · Staff Software Engineer, ML Observability

Role Posting Log in to save

As a Staff Software Engineer - ML Observability, you will join the ML Observability team to build tools that monitor, explain, and improve AI systems in production. You will focus on providing robust, scalable observability for AI workloads, specifically targeting Large Language Models and generative AI through features like drift detection, model evaluation, and behavior tracing. Your daily responsibilities include leading the design and implementation of new features, prototyping scalable products, and collaborating cross-functionally with engineering, product, UX, and applied science teams. You will leverage your expertise in distributed systems, backend architectures, inference pipelines, and prompt engineering to solve complex problems. The role requires proficiency in building LLM-powered applications and using observability platforms to help customers troubleshoot and optimize their AI systems within a fast-moving technical landscape.

What does a Software Engineer earn?

Median $197500 from 2196 postings across 133 companies.

See salary data

Skills

LLM Generative AI Machine Learning Observability Distributed Systems Backend Architecture Prompt Engineering Model Evaluation Tracing Inference Pipelines Software Engineering

What you'll do

  • Design and implement new features for LLM observability products.
  • Prototype and scale product features to provide insights for generative AI systems.
  • Develop tools for tracing, evaluating, and debugging Large Language Models.
  • Influence architecture decisions to build resilient, high-performance systems.
  • Mentor other engineers on technical execution and system design.
  • Translate customer pain points into product and engineering priorities.
  • Monitor industry trends in machine learning to drive team innovation.

What we're looking for

  • Hold a BS, MS, or PhD in Computer Science, Engineering, or a related scientific field (or equivalent experience).
  • Possess a deep understanding of distributed systems and scalable backend architectures.
  • Have hands-on experience building and shipping LLM-powered or Generative AI applications.
  • Understand model internals, inference pipelines, evaluation techniques, and prompt engineering.
  • Experience with observability tools and platforms.
  • Ability to communicate clearly, think rigorously, and write clean, maintainable code.
  • Demonstrate a product-oriented mindset and the ability to thrive in ambiguous, fast-changing environments.

Related searches

More Staff Software Engineer jobs All jobs at Datadog

More like this

Similar roles

Staff Software Engineer, Observability

Pinterest

Remote 7 days ago $177,185–$364,795
Distributed Systems Data Engineering Observability OpenTelemetry Prometheus Grafana Kafka Flink Java Python Go Scala Kubernetes Service Mesh Time-series Databases Columnar Storage Stream Processing Machine Learning Anomaly Detection
7+ yrs exp Remote
Save

Staff Software Engineer, Observability

Robinhood

Menlo Park, CA 44 days ago $230,000–$270,000
Kubernetes AWS Go Python Prometheus Grafana Honeycomb Humio Sentry Vector Distributed Systems Metrics Logs Tracing Alerting SLOs
8+ yrs exp Hybrid
Save

Senior ML/AI Engineer, Observability

General Motors (GM)

Sunnyvale, CA 73 days ago $178,420–$230,500
Go Kubernetes Docker Terraform Prometheus Grafana Istio GCP AWS Azure Unix Linux SSH Observability SLIs SLOs TSDBs
5+ yrs exp Hybrid
Save

Software Engineer, Observability

Apple Inc

Seattle, WA 24 days ago $175,000–$263,300
Time Series Database Distributed Systems Kubernetes Terraform CI/CD OpenTelemetry Prometheus Grafana Rust Python Go Java Scala GitOps Infrastructure as Code Query Engines Columnar Storage Configuration Management
5+ yrs exp
Save