below marketmost similar roles pay hereabove market
This role pays more than
72%
of similar roles. Most pay
$194,846–$253,812
— the blue band above.
At the midpoint, this role pays about
$249k
versus about
$224k
for comparable roles.
Based on 240 similar postings.
Employer
About Cisco
Cisco Systems is the world''s leading networking technology company, designing and manufacturing networking hardware, telecommunications equipment, and cybersecurity solutions for businesses and governments. Industry: Networking Technology & Cybersecurity
Cisco currently has
307 open roles
on FindRole.
Listed pay typically runs
$167,700–$245,000
across 224 roles with salary data.
The Machine Learning Engineering Technical Leader joins the Canvas AI team to establish the multi-quarter strategy for measuring agentic AI quality. This role involves owning the evaluation roadmap from architecture through production, where you will build layered systems including automated checks, calibrated LLM judges, and expert human reviews. You will design evaluation architectures for deterministic checks, model-based pre-release testing, and production traffic scoring to detect drift and regressions. Key responsibilities include mentoring engineers, driving shared measurement standards, and managing simulation and synthetic data generation. The role requires proficiency in Python and deep learning frameworks like PyTorch or TensorFlow. You will solve technical problems regarding multi-step reasoning, tool selection, and trajectory quality within an agentic workspace designed to help IT teams resolve issues across various domains.
Own the multi-quarter strategy and roadmap for measuring agentic AI quality from architecture to production.
Design a layered evaluation architecture including deterministic checks, model-based evaluations, and human expert calibration.
Build and calibrate multi-model LLM-as-judge panels to track true-positive and true-negative rates.
Manage a human-in-the-loop flywheel to convert SME annotations into training signals for automated judges.
Score production traffic online to detect drift and convert real traces into offline test cases.
Lead simulation, synthetic data generation, and adversarial testing for domains where ground truth is scarce.
Establish common standards, data contracts, and judge criteria across various partner teams.
Mentor engineers on evaluation engineering standards and represent the domain to senior leadership.
What we're looking for
BA/BS with 8+ years, MS with 6+ years, or PhD with 3+ years of experience in Computer Science, Machine Learning, or a related quantitative field.
Experience taking machine learning or LLM-based systems into production and owning them operationally.
Experience in Python and modern deep learning frameworks such as PyTorch or TensorFlow.
Experience designing evaluation, measurement, or quality systems for ML or LLM products, including datasets, metrics, judges, benchmarks, or regression suites.
Experience setting technical direction across multiple teams through influence rather than reporting lines.
LLM-as-judge design and calibration, including building judge systems and validating them against human experts (preferred).
Agentic system evaluation, including measuring multi-step reasoning, tool selection, and trajectory quality (preferred).
Production monitoring at scale, simulation and synthetic environments, and networking or infrastructure domain knowledge (preferred).