As a Distinguished Engineer on our Foundation Model Hosting team, you will lead the technical strategy and roadmap for serving large language models at scale, focusing on high throughput, low latency, and efficient GPU utilization. You will collaborate closely with AI Research & Science teams to co-design model architectures that are deployable in production environments, leveraging advanced techniques like speculative decoding and custom quantization strategies. Your responsibilities include mentoring senior engineers, establishing rigorous engineering standards, and fostering a culture of technical excellence. Ideal candidates have extensive experience in software engineering, particularly with languages such as Python or Go, and possess deep expertise in distributed systems, model optimization, and AI infrastructure projects like vLLM or TensorRT-LLM. This role demands contributions to open-source initiatives, published research at top-tier conferences, and hands-on experience optimizing NCCL and network fabrics for complex parallel architectures.
Skills
Python
Go
Java
JavaScript
TypeScript
Kubernetes
TensorRT-LLM
huggingface_TGI
Ray
Triton_Inference_Server
CUDA
NCCL
InfiniBand
RDMA
BGP
Wi-Fi
SD-WAN
Cloud_Networking
Data_Center_Networking
CI/CD
What you'll do
Design and drive the long-term technical roadmap for Foundation Model Hosting platform.
Lead performance engineering by implementing advanced techniques like speculative decoding and custom quantization strategies.
Act as primary engineering counterpart to AI Research & Science teams for model deployability.
Mentor senior engineers and establish rigorous engineering standards for AI deployment.
Foster a culture of uncompromising technical excellence in the team.
What we're looking for
At least 7 years of software engineering experience
Extensive experience in system development lifecycle and distributed inference communication
Contributions to open-source AI infrastructure or serving projects
Experience optimizing GPU utilization and network fabrics for complex parallelism architectures
Published research at top-tier ML and Systems conferences or hold related patents
Expertise in designing routing and scheduling mechanisms for multi-model serving environments
Market check
Salary context
This
$244,700–$279,200
range sits above
87%
of similar postings on FindRole.
Peer median band
$125,000–$202,346
Median floor and ceiling across peers.
Typical midpoint (25–75%)
$129,813–$223,400
Middle half of comparable postings.
Based on 238 comparable postings.*
* 240 is the maximum number of comparable postings sampled.
Employer
About Capital One Financial
Capital One Financial is a bank holding company specializing in credit cards, auto loans, banking, and savings products, known for its data-driven approach to consumer and commercial finance. Industry: Financial Services & Banking
Capital One Financial currently has
489 open roles
on FindRole.
Listed pay typically runs
$197,300–$225,100
across 483 roles with salary data.