- Full-time
- Palo Alto, United States
Senior AI Research Engineer - LLM Training & Inference Infrastructure
- Full-time
- New York, United States
As a Research Engineer on Datadog’s Frontier Foundations team, you will partner with Research Scientists to build and operate distributed systems for large-scale LLM training, reinforcement learning, and inference. You will develop infrastructure across GPU clusters that enables researchers to iterate quickly and complete long training runs reliably. The team supports research into autonomous agents for observability and builds the infrastructure needed to train and serve models at scale.
Requirements
- Strong expertise in distributed systems and ML infrastructure for large-scale training and inference; experience with Ray, Kubernetes, Slurm, or similar tools is a plus.
- Proficiency in Python and at least one systems language such as Rust, C++, or Go.
- Experience operating large-scale training or serving infrastructure, or schedulers and orchestration systems for GPU or other high-performance workloads.
- Understanding of distributed inference performance limits, including communication, memory, scheduling, and stragglers.
- Strong debugging, profiling, and incident-diagnosis skills; on-call experience is desirable.
- Ability to communicate design and performance trade-offs clearly to technical and non-technical audiences.
- Prior model-training experience is valuable but not essential for candidates with strong distributed-systems or HPC backgrounds.
- Design, build, and operate distributed training and inference infrastructure for LLMs, including Mixture-of-Experts models; scale training using parallelism, memory optimization, checkpointing, and failure recovery.
- Build inference, rollout, evaluation, experimentation, and RL systems, including sandboxes for online RL and agent post-training, without slowing training.
- Continuously benchmark rollout speed, training throughput, and learning behavior, and provide dashboards that reveal regressions.
- Improve researcher workflows for job submission, scheduling, quotas, GPU utilization, and observability.
- Collaborate with platform teams on cluster capacity, networking, and scheduling across GPU providers; investigate performance and reliability issues across hardware, software, and distributed systems.
- Additional advantages include experience with Megatron-LM, TorchTitan, DeepSpeed, vLLM, SGLang, RL systems, NCCL, RDMA, low-latency networking, GPU benchmarking and monitoring, sandboxed execution, evaluation infrastructure, or open-source ML and distributed-systems projects.
Additional information
Datadog is an observability and security platform company, and welcomes applicants who may not meet every qualification but are motivated to grow. Benefits may vary by country and employment type and include global benefits, RSUs, an employee stock purchase plan, collaboration with colleagues in New York City and Paris, conference opportunities, mentoring, and employee resource groups. Datadog is an equal-opportunity employer and provides an accessibility accommodation form for application assistance. Candidate information is handled under its privacy notice; its AI interviewing guidelines are available online.
Job seeker sign in
Please sign in to save this job.
By signing in, you agree to OnlyAIJobs’s Terms & Conditions and Privacy Policy in relation to your privacy information.
Don't have an account?
More jobs within the Development, Infrastructure and Research categories
- Full-time
- Dordrecht, Netherlands
Salary: 5100 - 6950 EUR MONTH
- Full-time
- Heerlen, Netherlands
Salary: 5533.19 - 7143.86 EUR P1M
AI
- Full-time
- San Francisco, United States
- Full-time
- London, United Kingdom
AI
- Full-time
- Oostrum, Netherlands
Salary: 3730 - 3730 EUR MONTH
Companies hiring AI talent
From research labs to fast-growing startups, these employers are building with Artificial Intelligence and recruiting on OnlyAIJobs.