AI Research Engineer - Datadog AI Research (DAIR) | Datadog Careers

What you'll do

• Build and operate multimodal data pipelines for training and evaluation.
• Implement models, run large-scale experiments, and optimize for performance and cost.
• Construct simulation environments and replay infrastructure for agent training.
• Orchestrate distributed training and RL with Ray, managing scaling and recovery.
• Establish automated benchmarks and regression tests for model predictions and agent performance.
• Collaborate on integrating research capabilities into Datadog’s products and services.

What we're looking for

• Extensive experience in distributed computing, reinforcement learning infrastructure, and large-scale ML systems.
• Proficiency in Python and familiarity with a systems language (e.g., Rust, C++, Go), plus cloud and data infrastructure knowledge.
• Practical experience implementing and operating ML training and inference systems at scale using frameworks like PyTorch or JAX.
• Hands-on experience with large-scale model training techniques including SFT, RLVR, RLHF, and efficient inference methods.
• Strong ability to explain technical design and performance trade-offs to both technical and non-technical audiences.
• Experience supporting research publications in top-tier conferences (e.g., NeurIPS, ICLR, ICML).
• Bonus: Software engineering skills in observability, SRE, or security domains.