Machine Learning - Data Scientist

Develop robust methodologies to assess the performance of foundation models (e.g., LLMs, vision-language models, etc.) across diverse tasks. Leverage LLMs as judges to perform subjective and open-ended model evaluations (e.g., for summarization, reasoning, or multimodal generation tasks). Build, curate, and lead evaluation datasets and benchmarks. Advanced proficiency in at least one scripting language, preferably Python. Collaborate with research, engineering, and product teams to define evaluation goals aligned with user experience and product quality. Conduct failure analysis and uncover edge cases to improve model robustness. Contribute to our tools and infrastructure to automate and scale evaluation processes. BS and a minimum of 3 years relevant industry experience Strong experience in evaluating supervised, unsupervised, and deep learning models. Hands-on experience evaluating LLMs and using them as scoring/judging mechanisms. Familiarity with multimodal models (e.g., image + text, video + audio) and related evaluation challenges. Proficiency in Python and libraries such as NumPy, pandas, scikit-learn, PyTorch, or TensorFlow. Solid understanding of statistical testing, sampling, confidence intervals, and metrics (e.g., precision/recall, BLEU, ROUGE, FID, etc.). Strong documentation skills, including the ability to write technical reports and present to non-technical audiences. Experience working with open-source evaluation tools like OpenEval, ELO-based ranking, or LLM-as-a-Judge frameworks. Familiarity with prompt engineering, few-shot or zero-shot evaluation techniques. Experience evaluating generative models (e.g., text generation, image generation). Prior contributions to ML benchmarks or public evaluations. Strong interpersonal skills.

For more details click Job Post.