Senior AI Engineer, APM Experiences

Datadog

Hybrid

Quick summary

Work type: Hybrid
Location: —
Salary: $187,000–$240,000 / yr
Posted: 25 days ago

Market check

Salary context

Competitive pay

How this pay compares to similar roles

Similar $212k

This role $214k

$162k most similar roles pay here $270k

This role pays more than 52% of similar roles. Most pay $176,937–$246,150 — the shaded band above. At the midpoint, this role pays about $214k versus about $212k for comparable roles.

Based on 240 similar postings.

Employer

About Datadog

Datadog, Inc. is an American company that provides an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform.

Datadog currently has 130 open roles on FindRole.

Listed pay typically runs $187,000–$240,000 across 65 roles with salary data.

Most-posted roles

View all roles at Datadog

At a glance

TL;DR · Senior AI Engineer, APM Experiences

Role Posting Log in to save

Join Datadog’s APM Experiences team as a senior engineer to lead the development of AI-powered features for Application Performance Monitoring, focusing on debugging and preventing performance issues. You will design and implement LLM- and Agent-based workflows that analyze telemetry data to provide diagnoses and optimizations, ensuring seamless integration with Datadog’s platform through Trace Explorer and Service Catalog. This role requires expertise in backend systems, real-time ML, and distributed tracing stacks, along with proficiency in Go, Java, or Python. Ideal candidates have 4+ years of experience delivering LLM features to production and a strong grasp of the ML lifecycle and observability practices.

Skills

Python Go Java Kubernetes Docker CI/CD Prometheus PostgreSQL AWS Grafana OpenTelemetry Datadog_APM SLO_SLIs Microservices LLM Agent_Frameworks RAG Observability_Data_Retrieval_Indexing

What you'll do

Shape AI experiences for APM by designing and shipping LLM/agentic workflows.
Own the full development loop from prototyping to productionizing scalable systems.
Build robust agent systems including tools, retrieval strategies, and guardrails.
Integrate new features with Datadog’s platform surfaces like Trace Explorer.
Write performant backend code and improve reliability for data systems.

What we're looking for

4+ years of experience building backend or real-time ML systems.
Proven track record delivering LLM/agent features to production.
Strong understanding of the ML lifecycle and statistical methods.
Expertise in choosing appropriate machine learning techniques for tasks.
Experience with microservices performance, tracing, and observability.
Proficiency in Go, Java, or Python; strong API/service design skills.

Similar roles

Senior Applied AI Engineer

Equifax

Alpharetta, GA 48 days ago

GCP Kubernetes Terraform Java Spring Boot Apache Beam Bigtable BigQuery PubSub GCS Composer Airflow Jenkins Helm CI/CD Python SQL NLP LLM CRMs APIs Event-driven architectures MLOps Docker Prometheus Grafana

Hybrid

Save

Senior AI Engineer

Intapp

Palo Alto, California +2 74 days ago

Python LangGraph LangChain Azure AWS Google Cloud Docker PostgreSQL Vector CI/CD Github Actions Azure Pipelines

Hybrid

Save

Senior AI Engineer

Morgan Stanley

New York, NY 53 days ago $120,000–$165,000

Python FastAPI Flask SQL Kubernetes Docker OpenTelemetry Grafana Prometheus CI/CD Jenkins GitHub Actions Redis Kafka

Save

Senior AI Engineer

Allstate

Remote 45 days ago $100,000–$170,500

Python RDF OWL SPARQL LLM Google ADK Microsoft Fabric Azure CI/CD MLOps Docker

Remote

Save

Senior AI Engineer

TSC

Dahlgren, VA 32 days ago

OWL RDF SPARQL SHACL Stardog Neo4j Amazon Neptune Python TensorFlow PyTorch rdflib owlready2 Apache NiFi Airflow FastAPI Flask React Angular Docker Kubernetes RFID

Save

Senior AI Engineer

Bristol Myers Squibb

Brisbane, CA +2 17 days ago $151,280–$183,319

Python AWS Azure OpenAI AWS Bedrock GitHub MCP context engineering vector databases Terraform CloudFormation Databricks MLOps DevOps agile CI/CD

Hybrid

Save