Lead Software Development Engineer in Test, AI Tooling & Automation

What you'll do

• Design, build, and maintain scalable, AI-first testing frameworks and automation suites for complex workflows.
• Translate high-level quality strategies into actionable plans focused on LLM integration and prompt accuracy.
• Perform white-box testing by analyzing application code to identify gaps and fix test flakiness.
• Implement evaluation processes for subjective AI dimensions like response relevance, safety, and contextual accuracy.
• Partner with cross-functional teams to integrate quality standards throughout the agile delivery lifecycle.
• Develop automated solutions within a fast-paced CI/CD environment to support rapid deployment schedules.

What we're looking for

• Strong technical fundamentals in Java, specifically regarding data structures and algorithms.
• Experience evaluating Large Language Models (LLMs) for quality response and familiarity with AI evaluation frameworks.
• Proven track record of delivering automation solutions in a fast-paced CI/CD environment.
• Ability to perform white-box testing and actively contribute to fixing code and reducing test flakiness.
• Strong analytical and problem-solving skills focused on identifying quality gaps and driving improvements.
• Excellent communication skills for cross-functional collaboration with product managers, engineers, and data scientists.
• Experience with public cloud infrastructure (AWS/Azure/GCP) or foundational knowledge of the Salesforce platform is preferred.
• Prior experience testing autonomous agents, conversational UIs, or generative AI workflows is a plus.