Senior Machine Learning Engineer, Speech LLM Evaluation

What you'll do

• Design and curate audio evaluation datasets that reflect real-world, multilingual, and conversational scenarios.
• Define and implement metrics for speech LLMs covering accuracy, robustness, and conversational quality.
• Build automated evaluation pipelines and "LLM-as-judge" tooling to scale model assessment.
• Analyze evaluation results to identify performance gaps and regressions for the modeling teams.
• Design rating protocols for human evaluation programs to validate automated metric reliability.
• Integrate new evaluation sets and metrics into shared infrastructure tools.
• Develop evaluation methodologies for emerging audio LLM capabilities and novel model behaviors.

What we're looking for

• Bachelor's degree in Computer Science, Electrical Engineering, or a related field, or equivalent practical experience.
• Experience building or working with text, speech, or audio evaluation pipelines and metrics.
• Proficiency in Python and experience building data processing pipelines at scale.
• Experience curating or annotating datasets for machine learning evaluation or training.
• Working knowledge of statistics as applied to measuring model performance and interpreting results.
• Familiarity with large language model evaluation techniques, including automated (LLM-as-judge) and human evaluation methods.
• Strong written and verbal communication skills to explain results to technical and non-technical audiences.
• Experience evaluating audio-native or multimodal models, designing human studies, or using distributed data frameworks (preferred).