Senior AI Engineer - APM Experiences | Datadog Careers

Datadog

Hybrid

Quick summary

Work type: Hybrid
Location: New York, NY
Salary: $187,000–$240,000 / yr
Posted: 15 days ago
Nearby: 99+ roles within 25 mi

Market check

Salary context

Competitive pay

How this pay compares to similar roles

Similar $216k

This role $214k

$167k most similar roles pay here $271k

This role pays more than 50% of similar roles. Most pay $184,975–$246,150 — the shaded band above. At the midpoint, this role pays about $214k versus about $216k for comparable roles.

Based on 240 similar postings.

Employer

About Datadog

Datadog, Inc. is an American company that provides an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform.

Datadog currently has 109 open roles on FindRole.

Listed pay typically runs $187,000–$240,000 across 58 roles with salary data.

Most-posted roles

View all roles at Datadog

At a glance

TL;DR · Senior AI Engineer - APM Experiences | Datadog Careers

Apply Now Log in to save

As a senior engineer on Datadog’s APM Experiences team, you will lead the development of AI-powered features that enhance application performance monitoring by detecting and resolving issues autonomously. Your day-to-day responsibilities include designing LLM and agent-based workflows to analyze telemetry data for diagnoses and optimizations, integrating these systems with Datadog’s platform, and ensuring robustness through reliable backend code and production ops. You will leverage technologies such as Go, Java, or Python, and have experience in distributed tracing, microservices performance, and observability. Ideal candidates possess a strong background in real-time ML systems, AI coding tools, and the ability to push boundaries in how AI can improve software engineering practices. This role offers significant influence on product direction and collaboration with skilled teammates to deliver impactful solutions for software engineers globally.

Skills

Python Go Java ML AI LLM Distributed_Tracing Microservices Observability API_Design Prometheus Grafana OpenTelemetry SLO_SLA Incident_Response CI/CD

What you'll do

Shape AI experiences for APM by designing and shipping LLM/agentic workflows.
Own the full development loop from prototyping to productionizing scalable systems.
Build robust agent systems including tools, retrieval strategies, and guardrails.
Integrate AI features with Datadog’s platform surfaces like Trace Explorer and monitors.
Write performant backend code and improve reliability for high-throughput data systems.

What we're looking for

4+ years of experience building backend or real-time ML systems.
Proven track record delivering LLM/agent features to production.
Ability to own user journeys and measure impact with clear product metrics.
Strong understanding of the ML lifecycle, including task definition and evaluation.
Experience with microservices performance and observability tools like tracing stacks.
Proficient in Go, Java, or Python; strong API/service design skills.

Similar roles

Senior AI Engineer | Datadog Careers

Datadog

New York, NY 15 days ago

Python Go Java LLMs Large Language Models Generative AI CI/CD Docker Kubernetes AWS PostgreSQL Git GitHub IDEs Static Code Analysis Compilers Dynamic Instrumentation

Hybrid

Save

Senior AI Platform Engineer- Data and Systems

Adobe

San Jose 51 days ago $208,300–$301,600

Apache_Spark Databricks Delta_Lake Kafka Kinesis Flink Python Scala SQL AWS Azure Docker Kubernetes CI/CD MCP LangChain LLMs Feature_Stores RAG Unity_Catalog FAISS Pinecone Weaviate Semantic_layers DataHub OpenMetadata AI-powered_developer_tools

Save

Senior Consultant - AI Engineering

IBM

Remote (Santa Clara, California, US) today

Python TensorFlow Kubernetes AWS Docker CI/CD Git PostgreSQL MongoDB Scikit-learn Pandas NumPy Jupyter Notebook Prometheus Grafana

Remote

Save

Senior Consultant - AI Engineering

IBM

Remote (Houston, Texas, US) today

Python TensorFlow Kubernetes AWS Docker CI/CD Git PostgreSQL MongoDB Scikit-learn Pandas NumPy Jupyter GitHub Swagger RESTful APIs Microservices MLOps

Remote

Save

Senior Consultant - AI Engineering

IBM

Remote (Capitola, California, US) today

Python TensorFlow Kubernetes Docker AWS CI/CD Git PostgreSQL MongoDB Scikit-learn Pandas NumPy Jupyter Notebook Swagger RESTful API

Remote

Save

Senior Consultant - AI Engineering

IBM

Remote (Irvine, California, US) today

Python TensorFlow Kubernetes AWS Docker CI/CD Git PostgreSQL MongoDB Scikit-learn Pandas NumPy Jupyter Notebook Swagger RESTful API

Remote

Save