Senior Software Development Engineer in Test, LLM Evaluation & Automation

What you'll do

• Design and maintain LLM-as-a-judge evaluation harnesses integrated into CI and automation pipelines.
• Author and curate evaluation sets and rubrics to judge model outputs at scale.
• Run large-scale evaluation jobs and triage results to distinguish model regressions from infrastructure noise.
• Build test coverage ranging from low-level component tests to full end-to-end user flows.
• Package evaluation tooling into reusable libraries and jobs for adoption by other engineering teams.
• Define quality gates and reporting systems to catch model regressions before human evaluation or rollout.
• Collaborate with data scientists to develop methods for spotting regressions across large output sets.

What we're looking for

• BS in Computer Science, Mathematics, or a related field (or equivalent practical experience).
• Three years of relevant industry experience in test automation, software development, or related areas.
• Strong practical knowledge of Python, including data-pipeline fluency with JSON, YAML, and REST APIs (preferred).
• Hands-on experience with LLM-as-a-judge evaluation and rubric design (preferred).
• Strong software engineering fundamentals to build maintainable pipelines and tooling rather than just scripts (preferred).
• Strong knowledge of the software development lifecycle, testing methodologies, and QA processes (preferred).
• Experience building on-device tooling and device/model evaluation infrastructure (preferred).
• Experience with Xcode (preferred).