Lead Software Engineer, Machine Learning Platform

What you'll do

• Design, build, and maintain an end-to-end ML training platform.
• Optimize GPU training workloads for throughput, utilization, and reproducibility across single-node and distributed systems.
• Build and operate training infrastructure on Kubernetes platforms including resource management and troubleshooting.
• Enable Gen AI and LLM training workflows including supervised fine-tuning and evaluation harnesses.
• Implement observability for training systems through metrics, logs, dashboards, and operational runbooks.
• Improve developer experience by providing standardized containers, CI/CD pipelines, and self-service workflows.
• Drive team adoption of AI-assisted engineering practices to improve code quality and delivery speed.
• Optimize training costs and performance through right-sizing, scheduling policies, and quota planning.

What we're looking for

• Formal training or certification on software engineering concepts and 5+ years of applied experience.
• Demonstrated experience running ML training in cloud environments and debugging issues across infrastructure and code.
• Strong Python skills with solid engineering practices including testing, code reviews, modular design, and dependency management.
• Hands-on experience with deep learning training workflows and at least one major framework like PyTorch or TensorFlow.
• Experience with distributed training concepts such as DDP, FSDP, DeepSpeed, and collective communication basics.
• Experience with Kubernetes fundamentals and AWS services including EKS, ECR, S3, IAM, and VPC.
• Demonstrated experience leading the use of approved AI-assisted software development tools and coaching engineers on responsible AI use.
• Experience running training workloads across multiple cloud platforms (preferred).