Lead Software Engineer, Machine Learning

What you'll do

• Design, build, and maintain an end-to-end ML training platform.
• Optimize GPU training workloads for throughput, utilization, and reproducibility across single-node and distributed systems.
• Build and operate training infrastructure on Kubernetes, including resource management and workload troubleshooting.
• Enable Gen AI and LLM training workflows, including supervised fine-tuning and evaluation harnesses.
• Implement observability for training systems using metrics, logs, dashboards, and operational runbooks.
• Improve developer experience through standardized containers, CI/CD pipelines, and self-service workflows.
• Drive team adoption of AI-assisted engineering practices to improve code quality and delivery speed.
• Optimize training costs and performance through right-sizing, scheduling policies, and quota planning.

What we're looking for

• Formal training or certification on software engineering concepts and 5+ years of applied experience.
• Demonstrated experience running ML training in cloud environments and debugging issues across infrastructure and code.
• Strong Python skills with solid engineering practices including testing, code reviews, modular design, and dependency management.
• Experience building automation and CI for ML codebases including build, test, release, and deployment workflows.
• Hands-on experience with deep learning training workflows and at least one major framework such as PyTorch or TensorFlow.
• Experience with distributed training concepts including DDP, FSDP, DeepSpeed, and collective communication basics.
• Experience with Kubernetes fundamentals for compute-intensive workloads and AWS services like EKS, S3, IAM, and VPC.
• Demonstrated experience leading the use of approved AI-assisted software development tools and coaching engineers on responsible AI use.