This role pays more than
65%
of similar roles. Most pay
$162,000–$223,117
— the shaded band above.
At the midpoint, this role pays about
$209k
versus about
$193k
for comparable roles.
Based on 240 similar postings.
Employer
About Microsoft
Microsoft Corporation is a global technology leader producing software, hardware, and cloud services including Windows, Office 365, Azure cloud platform, Xbox gaming, and Surface devices. Industry: Software & Cloud Computing
Microsoft currently has
598 open roles
on FindRole.
Listed pay typically runs
$119,800–$234,700
across 586 roles with salary data.
As a Principal Security Research Manager on the AI Evaluation Systems team, you will build the common evaluation platform and methodology used by security agent programs. You will design and build end-to-end evaluation harnesses for agentic security workflows, including triage, remediation, and tool use. Your daily work involves creating benchmark suites, implementing deterministic validators, automated graders, and trace analyzers to measure task success, safety, and cost tradeoffs. You will manage the development of infrastructure for tracking metrics like latency and token usage while integrating evaluations into CI/CD pipelines and release gates. The role requires expertise in LLMs, AI agents, prompt orchestration, and security engineering. You will solve the critical problem of establishing a trust system to determine when agentic systems are safe, reliable, and ready for production use in complex security environments.
Design and build end-to-end evaluation harnesses for agentic security and engineering workflows.
Create representative benchmark suites and golden datasets including edge cases and adversarial scenarios.
Implement deterministic validators, automated graders, and trace analyzers to make evaluations repeatable.
Measure performance metrics including task success, safety compliance, latency, cost, and human-review effort.
Compare models, prompts, and orchestration patterns to identify quality versus efficiency tradeoffs.
Integrate evaluation results into CI/CD pipelines and release gates to provide evidence for production decisions.
Connect offline evaluation data with real-world production feedback and telemetry to improve benchmark coverage.
Develop reusable paved paths including playbooks, templates, and dashboards to scale across the organization.
What we're looking for
A Bachelor's degree in Statistics, Mathematics, Computer Science, Computer Security, or a related field is required with 6+ years of relevant experience.
A Master's degree in Statistics, Mathematics, Computer Science, Computer Security, or a related field is required with 4+ years of relevant experience.
A Doctorate in Statistics, Mathematics, Computer Science, Computer Security, or a related field is required with 3+ years of relevant experience.
At least 1 year of people management experience is required.
Proven software engineering experience building production systems, developer platforms, test infrastructure, automation frameworks, and data pipelines.
Experience with LLMs, AI agents, model evaluation, prompt/tool orchestration, automated grading, and benchmark design.
Experience with experimentation, statistical confidence, evaluator calibration, regression analysis, and quality measurement systems.
Experience in security engineering, vulnerability management, SAST/SCA, remediation workflows, or compliance-sensitive systems.
AI
Cybersecurity
Threat Analysis
Vulnerability Research
Anomaly Detection
Software Development Lifecycle
Data Analysis
Security Telemetry
Automation
Observability
Large-scale Computing