Lead Software Development Engineer in Test

What you'll do

• Analyze requirements to produce comprehensive test strategies and detailed test cases.
• Design and implement manual and automated tests across unit, component, integration, and end-to-end levels.
• Build and enhance scalable automated test frameworks for feature testing and AI-driven workflows.
• Develop and maintain automation for UI and API tests using Selenium, pytest, and Playwright.
• Conduct performance and load testing using tools like Locust to ensure product scalability.
• Evaluate AI agents for multi-step task completion, tool selection accuracy, and failure recovery.
• Build evaluation datasets and scoring methods using deterministic checks, human feedback, and LLM-based judges.
• Integrate evaluations into CI/CD pipelines and translate production failures into regression coverage.

What we're looking for

• Experience evaluating AI agents, including multi-step task completion, tool selection, tool-call accuracy, and failure recovery.
• Working knowledge of agent frameworks, preferably LangChain, with the ability to understand and debug context handling and tool execution (preferred).
• Hands-on experience with LangSmith or similar platforms for evaluation datasets, custom evaluators, experiments, and comparisons.
• Experience with agent observability, using traces and telemetry to investigate quality issues, latency, token usage, and cost.
• Strong Python software engineering skills to build and extend evaluation tooling, metrics pipelines, and reporting systems.
• Applied statistics and data analysis skills to assess variability, compare experiments, and distinguish meaningful improvements from noise.
• Experience developing evaluation datasets and scoring methods, combining deterministic checks, human feedback, and LLM-based judges.
• Export controls.