Principal Software Engineer – Large-Scale LLM Memory and Storage Systems

What you'll do

• Design a unified memory layer spanning multiple tiers for large-scale LLM inference.
• Implement deep integrations with leading LLM serving engines focusing on KV-cache management.
• Co-design interfaces enabling disaggregated prefill and peer-to-peer KV-cache sharing across clusters.
• Exploit GPUDirect, RDMA, NVLink technologies to ensure low-latency cache access in heterogeneous systems.
• Mentor engineers and set technical direction for memory and storage subsystems within the team.

What we're looking for

• 15+ years experience building large-scale distributed systems and high-performance storage infrastructure.
• Deep understanding of memory hierarchies including GPU HBM, host DRAM, SSD, and remote/object storage.
• Hands-on experience with networked I/O technologies like RDMA/NVMe-oF/NVLink for low-latency data access.
• Strong skills in profiling and optimizing systems across CPU, GPU, memory, and network layers.
• Experience designing unified memory or storage layers spanning multiple tiers for performance and cost efficiency.
• Prior contributions to open-source projects focused on KV-cache optimization and reuse in LLM serving systems.