SRE, AI Inference Engineer

What you'll do

• Build and maintain robust inference engines using tools like vLLM, TGI, and NVIDIA Triton.
• Optimize Large Language Models for diverse environments ranging from GPU data centers to edge devices.
• Profile and optimize models for specialized hardware backends including NVIDIA GPUs, Apple Silicon, TPUs, and LPUs.
• Design and implement auto-scaling architectures for real-time and batch inference pipelines using Kubernetes.
• Establish observability frameworks to monitor metrics like Time to First Token and tokens per second.
• Execute performance and load testing suites to identify bottlenecks and ensure system reliability at scale.
• Implement deployment optimizations to deliver low-latency AI serving solutions for multiple business applications.

What we're looking for

• Proficiency in programming languages such as Python, C++, Rust, or Golang for high-performance AI workflows.
• Proven hands-on experience with inference tools like vLLM, TensorRT, Llama.cpp, and Ollama.
• Strong familiarity with infrastructure technologies including Docker, Kubernetes, and cloud platforms like AWS, GCP, and Azure.
• Comprehensive understanding of GPU and AI hardware, including profiling and optimizing performance for NVIDIA GPUs and TPUs.
• Prior experience deploying LLMs with advanced techniques like Speculative Decoding or PagedAttention (preferred).
• Contributions to open-source inference libraries or hardware-level kernel development (e.g., CUDA, Triton kernels) (preferred).
• Background in MLOps or SRE roles focused on high-performance AI endpoints and reliability (preferred).
• Proficiency in designing scalable solutions for high-throughput inference environments optimized for traffic bursts (preferred).