Senior Software Development Engineer in Test, LLM Evaluation & Automation

What you'll do

• Design and maintain LLM-as-a-judge evaluation harnesses integrated into CI and automation pipelines.
• Author and curate evaluation sets and rubrics to judge model outputs at scale.
• Triage large-scale evaluation results to distinguish actual model regressions from infrastructure noise.
• Build test coverage ranging from low-level component tests to full end-to-end user flows.
• Develop reusable libraries and tools for evaluation tasks that other engineering teams can adopt.
• Define quality gates and reporting systems to catch regressions before human evaluation or public release.
• Create automated systems to evaluate image generation, natural language responses, and summarized content.

What we're looking for

• Bachelor of Science in Computer Science, Mathematics, or a related field (or equivalent practical experience).
• Three years of relevant industry experience in test automation, software development, or related areas.
• Proficiency in Python including data-pipeline fluency with JSON, YAML, and REST APIs.
• Experience with LLM-as-a-judge evaluation, rubric design, and generative model behavior (image/NLP).
• Strong software engineering fundamentals to build maintainable pipelines and reusable libraries rather than just scripts.
• Expertise in the software development lifecycle, testing methodologies, and QA processes.
• Ability to triage results, distinguish regressions from noise, and communicate quality signals to leadership.
• Experience with CI/CD integration, job orchestration systems, and database/query tooling like SQL.