Principal AI Architect

What you'll do

• Define the technical vision and architecture for AI, LLM, RAG, and agent evaluation systems across the product lifecycle.
• Design evaluation frameworks to assess product quality during development, at launch, and post-ship.
• Identify behavioral gaps, failure modes, and model limitations before they become customer-impacting issues.
• Integrate evaluation methods into build pipelines, release gates, and engineering workflows as a standard practice.
• Build systems that connect product telemetry, offline evaluation, human judgment, and automated quality signals.
• Translate ambiguous product goals into measurable evaluation strategies, success criteria, and technical plans.
• Drive architecture decisions for data pipelines, model interfaces, search systems, and reporting dashboards.
• Mentor senior engineers and applied scientists on building reliable and scalable evaluation infrastructure.

What we're looking for

• Bachelor's Degree in a relevant field and 6+ years experience, Master's and 4+ years, or Doctorate and 3+ years of related experience.
• Master's Degree in a relevant field and 9+ years experience, or Doctorate and 6+ years of related experience.
• 5+ years experience creating publications such as patents, libraries, or peer-reviewed academic papers.
• 5+ years experience conducting research as part of a research program in academic or industry settings.
• 2+ years experience presenting at conferences or other events as an invited speaker.
• 3+ years experience developing and deploying live production systems or products from ideation to shipping.
• Demonstrated experience evaluating LLMs, including designing eval datasets, defining quality metrics, and analyzing model behavior.
• Experience building evaluation platforms, experimentation systems, model observability, or agent evaluation infrastructure.