Senior Staff Engineer, ML Inference

What you'll do

• Architect and own high-performance production ML inference systems for high throughput and low latency.
• Optimize model performance using CUDA, TensorRT, Triton, TVM, and custom GPU kernels.
• Implement advanced techniques like pruning, quantization, distillation, and batching to improve serving efficiency.
• Drive cost optimization and system efficiency to reduce cloud spend and carbon footprint.
• Lead deep performance investigations to resolve bottlenecks in large-scale ML workloads.
• Define the technical strategy and culture for ML inference across the organization.
• Partner with cross-functional teams to deploy, benchmark, and scale cutting-edge models.

What we're looking for

• Proven expertise in building and optimizing large-scale ML inference systems with measurable performance wins.
• Experience in production model serving, runtime optimization, and acceleration using GPUs (CUDA, TensorRT).
• Strong software engineering skills in Python, C++, or other relevant languages with a distributed computing mindset.
• Demonstrated leadership in architecting and scaling reliable, real-time inference handling millions of queries daily.
• Advanced understanding of model compression techniques including pruning, quantization, distillation, and batching.
• Ability to collaborate cross-functionally with ML research, infrastructure, and product teams.
• Experience optimizing inference across diverse hardware such as NVIDIA, AMD, ARM, or cloud TPUs.
• Familiarity with MLOps pipelines, monitoring, observability, and auto-scaling for inference platforms.