Senior / Staff Machine Learning Engineer, ACAI Training & Inference Platform

What you'll do

• Design and build scalable distributed AI execution frameworks for large-scale model training and inference using Ray.
• Develop high-level APIs, SDKs, and reusable workflows for supervised fine-tuning, preference optimization, and reinforcement learning.
• Optimize LLM inference runtimes to improve throughput, reduce latency, and lower costs across diverse hardware configurations.
• Drive deep runtime optimizations for GPU execution, memory management, kernel utilization, and distributed inference.
• Evaluate and integrate modern inference and training engines like vLLM and SGLang into the AI platform.
• Design distributed execution strategies for large models, including parallelism, workload placement, and resource allocation.
• Improve system reliability through fault tolerance, checkpointing, workload recovery, and performance diagnostics.
• Establish performance benchmarks to quantify improvements in training efficiency and inference cost per token.

What we're looking for

• 6+ years of experience building distributed systems, AI platforms, model execution frameworks, or high-performance software systems.
• Hands-on experience in large-scale AI training, LLM post-training and reinforcement learning, or LLM inference and serving optimization.
• Proficiency in Python and experience with a systems-level language such as C++, Rust, Go, or Java.
• Experience with PyTorch and modern foundation model architectures, including the execution and performance requirements of large language models.
• Understanding of distributed execution and parallelism strategies for large models, including data, tensor, pipeline, or expert parallelism.
• Experience profiling and optimizing AI workloads across GPUs or other accelerators to identify and resolve performance bottlenecks.
• Experience building APIs, SDKs, or reusable software components that simplify AI model development, training, adaptation, evaluation, or deployment.
• Experience with Ray or comparable distributed execution frameworks and familiarity with production-scale AI workloads.
• B.S., M.S., or Ph.D. in Computer Science, Artificial Intelligence, or a related technical field, or comparable software engineering experience.
• Experience building end-to-end LLM post-training workflows, including SFT, preference optimization, RLHF, or RLVR (preferred).
• Experience with distributed training and post-training frameworks such as Ray Train, TRL, VeRL, RLlib, DeepSpeed, FSDP, or equivalent (preferred).
• Experience designing developer-facing training APIs or services that support iterative experimentation, checkpointing, evaluation, and scalable model adaptation (preferred).
• Deep understanding of LLM inference optimization, including continuous batching, KV-cache management, quantization, speculative decoding, prefix caching, and disaggregated prefill/decode (preferred).
• Experience optimizing inference engines such as vLLM, SGLang, TensorRT-LLM, or similar frameworks for latency, throughput, GPU utilization, and cost per token (preferred).
• Experience with deep runtime optimization, including GPU kernels, memory allocation, graph execution, communication primitives, scheduling, and accelerator utilization (preferred).
• Experience benchmarking and tuning inference workloads across model architectures, hardware configurations, concurrency levels, and latency targets (preferred).
• Understanding of LLM serving architectures, including dynamic batching, autoscaling, multi-model serving, heterogeneous hardware, and workload-aware resource allocation (preferred).
• Experience integrating training and inference engines into production AI platforms through stable APIs, reusable execution abstractions, and reliable operational workflows (preferred).
• Contributions to open-source AI training, post-training, distributed execution, inference, or model-serving projects (preferred).
• Familiarity with Ray Data, Spark, Flink, or similar data-processing technologies used in AI data preparation and training pipelines (preferred).
• Experience operating AI workloads in production across cloud or on-premises accelerator environments, including Kubernetes or comparable execution systems (preferred).