Senior Software Development Engineer in Test, LLM Evaluation & Automation

What you'll do

• Design and maintain LLM-as-a-judge evaluation harnesses integrated into CI and automation pipelines.
• Author and curate evaluation sets and rubrics to judge model outputs at scale.
• Triage large evaluation results to distinguish actual model regressions from infrastructure noise or rubric errors.
• Build test coverage ranging from low-level component tests to full end-to-end user flows for generative features.
• Develop reusable libraries and tools for evaluation tasks that can be adopted by other engineering teams.
• Establish quality gates and reporting systems to catch model regressions before human evaluation or public release.
• Create automated systems to evaluate image generation, natural language responses, and summarized content.

What we're looking for

• BS in Computer Science, Mathematics, or a related field (or equivalent practical experience).
• Three years of relevant industry experience in test automation, software development, or related areas.
• Strong practical knowledge of Python and data-pipeline fluency including JSON, YAML, and REST APIs.
• Hands-on experience with LLM-as-a-judge evaluation and rubric design.
• Strong software engineering fundamentals to build maintainable pipelines and tooling rather than just scripts.
• Experience integrating with CI/CD systems and deploying tools as reusable libraries.
• Familiarity with generative model behavior including image generation, NLP, or LLM output evaluation.
• Experience with database/query tooling such as SQL and dashboards for reporting quality trends.