As a Senior AI Software Architect in the SPARC team within Microsoft’s Azure Hardware Systems and Infrastructure (AHSI) organization, you will play a pivotal role in enabling and optimizing large-scale AI models on Maia accelerators. Your daily tasks include porting and optimizing models using frameworks like PyTorch, ONNX, vLLM, and SGLang, applying quantization techniques such as BF16 to FP8 conversion for efficient inference, and experimenting with parallelism strategies across different interconnects. You will also collaborate closely with hardware architects and kernel developers to co-design solutions, ensuring models run efficiently on Maia hardware while working on improving the inference stack through performance tuning at the PyTorch level and assisting in kernel performance analysis. This role requires expertise in AI inference stacks, Triton kernels, CUDA programming, and a strong background in model optimization techniques, making it ideal for someone with a growth mindset and a passion for innovation in cloud infrastructure.
Port and optimize large-scale AI models to run efficiently on Maia hardware.
Apply quantization techniques like BF16 → FP8 for efficient inference and training.
Experiment with parallelism strategies (TP, PP) and analyze performance impacts across interconnects.
Collaborate on improving inference pipelines including KV caching in sglang/vllm.
Assist in kernel performance analysis and work with Triton kernels for basic operations.
What we're looking for
3+ years of hands-on experience with PyTorch and model optimization techniques.
Practical knowledge of quantization techniques like PTQ/QAT, especially for KV cache quantization.
Familiarity with parallelization strategies and distributed training concepts such as sharding and allreduce.
2+ years of experience with AI inference stacks like SGLang/vLLM and performance profiling.
Excellent problem-solving and communication skills; ability to work in a collaborative team environment.
Experience with Triton kernels and CUDA programming, willingness to learn essential.
3+ years of prior work on efficient model checkpointing, resharding scripts, and large-scale model deployments for serving at scale.
Market check
Salary context
This
$119,800–$234,700
range sits above
41%
of similar postings on FindRole.
Peer median band
$148,250–$241,500
Median floor and ceiling across peers.
Typical midpoint (25–75%)
$177,250–$235,750
Middle half of comparable postings.
Based on 240 comparable postings.*
* 240 is the maximum number of comparable postings sampled.
Employer
About Microsoft
Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing
Microsoft currently has
534 open roles
on FindRole.
Listed pay typically runs
$119,800–$234,700
across 488 roles with salary data.
Remote (Santa Clara, CA)13 days ago
$224,000–$356,500
C
C++
Rust
RDMA
GPUDirect
NVLink
InfiniBand
RoCE
GPU
DPU
NIC
switch
vLLM
SGLang
TensorRT-LLM
NVMe-oF
GPUDirect Storage
S3
Reinforcement Learning
ML inference frameworks