Software Engineer, Inference

What you'll do

• Develop high-throughput inference systems to serve AI models internally across SpaceX.
• Architect and implement scalable distributed infrastructure for model serving, including load balancing and auto-scaling.
• Optimize inference latency and throughput using low-level GPU kernels, quantization, and speculative decoding.
• Build high-concurrency serving systems with 100% uptime and low tail latency.
• Own end-to-end components such as request routing, SDK development, and rate limiting.
• Benchmark and accelerate inference engines like SGLang, vLLM, and TensorRT-LLM.
• Create custom tools for tracing, replaying, and resolving issues across the full stack.
• Build robust CI/CD infrastructure for seamless endpoint deployment and inference engine updates.

What we're looking for

• Bachelor's degree in computer science, engineering, math, or scientific discipline; OR 2+ years of professional software experience.
• Experience in designing, implementing, and maintaining reliable and horizontally scalable distributed systems.
• 1+ years of experience in full stack or backend development with production systems.
• 1+ years of experience with Rust or C++.
• Experience with LLM inference engines and serving frameworks (preferred).
• Deep low-level systems programming and optimizations including GPU kernels and quantization (preferred).
• Experience with Docker, Kubernetes, and containerized applications (preferred).
• Must be a U.S. citizen, national, lawful permanent resident, refugee, asylee, or eligible for Department of State authorization.