Senior Solutions Architect, First Time Deployment Validation

What you'll do

• Set up, adjust, and verify AI factory environments across multi-GPU and multi-node Linux clusters.
• Execute and analyze key AI/LLM benchmarks, including setup, orchestration, and result collection.
• Investigate and resolve issues when training jobs or benchmarks fail, hang, or underperform.
• Build and improve observability systems for metrics, logs, traces, and dashboards to monitor system health.
• Develop Python and Shell automation for running benchmarks, collecting results, and performing regression checks.
• Analyze NCCL communication patterns and collectives like AllReduce and AllToAll for AI/LLM workloads.
• Recommend configuration changes to job parallelism and cluster settings to improve throughput and scaling efficiency.
• Create documentation and readiness collateral to support internal collaborators and customer-facing teams.

What we're looking for

• Bachelor’s degree or equivalent experience in Computer Science, Mathematics, Engineering, Physics, or a related field.
• More than 6+ years of experience managing Linux-based systems in HPC, distributed systems, or extensive AI/ML settings.
• Hands-on experience running AI/ML workloads on multi-GPU and/or multi-node clusters with practical knowledge of NCCL.
• Solid grasp of collective communication patterns, particularly AllReduce and AllToAll, applied in contemporary ML/LLM training.
• Familiarity with LLM training and/or inference workflows using frameworks such as PyTorch or TensorFlow.
• Proficiency with Python and Shell/Bash for scripting, automation, and tooling.
• Experience with benchmarking, including crafting, executing, and interpreting performance benchmarks.
• Experience with AI factory or large-scale AI infrastructure build, deployment, or operations (preferred).