Senior Staff GenAI Engineer, Application Performance Monitoring (APM)

Datadog

Confirmed live 2 days ago High trust
Hybrid

Quick summary

Work type
Hybrid
Location
Salary
$272,000–$340,000 / yr
Posted
101 days ago
Freshness
Confirmed live 2 days ago

Market check

Salary context

Above market

How this pay compares to similar roles

Similar $211k
This role $306k
$156k most similar roles pay here $360k

This role pays more than 95% of similar roles. Most pay $176,193–$246,150 — the shaded band above. At the midpoint, this role pays about $306k versus about $211k for comparable roles.

Based on 240 similar postings.

Employer

About Datadog

Datadog, Inc. is an American company that provides an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform.

Datadog currently has 266 open roles on FindRole.

Listed pay typically runs $161,000–$205,000 across 138 roles with salary data.

Most-posted roles

View all roles at Datadog

At a glance

TL;DR · Senior Staff GenAI Engineer, Application Performance Monitoring (APM)

Senior Staff GenAI Engineer - Application Performance Monitoring (APM) is a technical leadership role within the APM team focused on agentic workflows and automated investigation tools. You will serve as the technical owner for GenAI initiatives, where you will design, train, evaluate, and deploy GenAI/ML models at scale to provide deeper visibility into application performance and troubleshooting. The role involves building and benchmarking models using state-of-the-art techniques while guiding long-term strategy across multiple teams. You will work with cross-functional teams to develop tools that solve complex problems in distributed tracing and service optimization. Required skills include extensive experience in model development, deployment, and evaluation, as well as proficiency in utilizing AI coding tools. This position addresses the technical challenge of creating agentic systems for identifying performance bottlenecks and triaging issues within complex software environments.

What you'll do

  • Serve as the technical owner for GenAI initiatives within the APM group.
  • Design, train, evaluate, and deploy GenAI/ML models at scale.
  • Define long-term strategy and technical direction for agentic workflows across products.
  • Build and benchmark GenAI/ML models using state-of-the-art techniques.
  • Develop automated investigation and triaging tools in collaboration with cross-functional teams.
  • Influence product direction by advocating for the end user through a product-minded approach.
  • Mentor engineers and lead technical design reviews to influence engineering culture.

What we're looking for

  • Hold a BS, MS, or PhD in a scientific field or equivalent experience.
  • Possess 10+ years of relevant engineering experience.
  • Have 4+ years of experience leading cross-team technical initiatives.
  • Demonstrate a proven track record of leading large-scale GenAI/ML initiatives in a product-driven environment.
  • Exhibit proficiency in model development, deployment, evaluation, and optimization for end-to-end ML systems.
  • Ability to drive initiatives across cross-functional teams and solve ambiguous challenges.
  • Experience using AI coding tools in daily workflows to validate and refine AI-generated output.
  • Proven ability to mentor engineers and cultivate a high-performing engineering culture.

More like this

Similar roles

Senior Lead AI Engineer, GenAI Platform Services

Capital One Financial

San Jose, CA +3 91 days ago $229,900$262,400
Python Go Scala Java C++ C# AWS Huggingface PyTorch VectorDBs LLM Inference Nemo Guardrails Machine Learning Similarity Search SaaS AI Model Evaluation Observability
6+ yrs exp

Senior Lead AI Engineer, GenAI Platform Services

Capital One Financial

New York, NY +3 22 days ago $250,800$286,200
Python Go Scala Java C++ C# AWS Huggingface PyTorch VectorDBs Nemo Guardrails LLM Inference Similarity Search Machine Learning GenAI SaaS
6+ yrs exp

Senior Lead AI Engineer, GenAI Platform Services

Capital One Financial

San Francisco, CA 11 days ago $229,900$262,400
Python Go Scala Java C++ C# AWS Huggingface PyTorch VectorDBs LLM Inference Nemo Guardrails Machine Learning GenAI Similarity Search
6+ yrs exp