This role pays more than
54%
of similar roles. Most pay
$166,375–$246,150
— the shaded band above.
At the midpoint, this role pays about
$209k
versus about
$206k
for comparable roles.
Based on 240 similar postings.
Employer
About Zillow
Zillow Group is a leading real estate and rental marketplace providing consumers with data, tools, and services to find, buy, sell, rent, and finance homes, and connecting buyers with agents and lenders. Industry: Real Estate Technology & Marketplace
Zillow currently has
23 open roles
on FindRole.
Listed pay typically runs
$148,600–$237,400
across 23 roles with salary data.
Agentic AI, Senior Software Development Engineer joins the Agentic Evals team to build essential infrastructure for real estate AI assistants. This role focuses on developing evaluation frameworks, tracing systems, observability tools, and feedback loops to ensure AI agents remain trustworthy and measurable. You will design and scale backend systems that allow internal teams to measure quality, reliability, and performance while managing costs and incident response. The position requires expertise in building scalable production ML services, platform infrastructure, and developer-facing tools. Key technologies include LLM orchestration tools like LangChain or LangGraph, and evaluation frameworks such as Databricks MLflow, LangSmith, Braintrust, or Promptfoo. You will solve the complex problem of quantifying AI behavior by translating emerging agentic paradigms into practical features, ensuring that automated systems provide reliable user experiences within the real estate domain through robust measurement and quality control.
What does a Software Development Engineer earn?
Median $194400 from 179 postings across 17 companies.
Design, build, and scale evaluation frameworks for Zillow's agentic AI experiences.
Build tracing, observability, and quality measurement systems for production AI agents.
Create platform infrastructure that allows internal domain teams to build on the Evals platform without bottlenecks.
Integrate new AI evaluation capabilities into production systems in partnership with applied scientists and machine learning engineers.
Develop methods to measure AI quality, reliability, trustworthiness, and overall product impact.
Translate emerging agentic AI paradigms and LLM tooling into practical platform innovations.
Manage the scaling, reliability, performance optimization, and cost management of the evaluation layer.
What we're looking for
You must have 4+ years of backend engineering experience in designing, shipping, and operating scalable production ML services.
You must have experience building platform infrastructure or developer-facing services consumed by multiple teams.
You must be comfortable with LLMs, agentic systems, and evaluation frameworks including orchestration tools like LangChain or LangGraph.
You should have hands-on experience with ML Evals & Observability frameworks (Databricks MLflow, LangSmith, Braintrust, Promptfoo, or similar) (preferred).
You must be able to translate technical concepts across applied science, ML, product, design, and engineering teams.
You should have an advanced degree in Computer Science, Machine Learning, or a related field (preferred).
You should have experience designing and operating evaluation pipelines including offline batch evals and online scoring (preferred).
You should have experience building products using frontier LLMs, multimodal models, or agent-based systems (preferred).