Senior Applied Scientist

What you'll do

• Develop and execute a comprehensive strategy for LLM evaluation including quality, cost, performance, and responsible AI.
• Build and maintain a robust measurement framework to track and report on LLM performance and user impact.
• Drive the engineering product roadmap to construct automated evaluation pipelines integrated into the product workflow.
• Use data science skills to design experiments, analyze data, and derive actionable insights to improve LLM systems.
• Lead efforts to assess and improve the effectiveness of language models and agents through synthetic and manufactured data.
• Implement Responsible AI principles to ensure ethical and unbiased practices in model development and deployment.
• Conduct deep research on LLM evaluation and user experience optimization to contribute to the scientific body.
• Communicate findings and recommendations to executive leadership to foster a data-driven culture.

What we're looking for

• Bachelor's degree plus 4 years experience, Master's plus 3 years, or Doctorate plus 1 year in a relevant technical field.
• Equivalent professional experience in statistics, predictive analytics, or research is acceptable.
• Ability to pass the Microsoft Cloud background check and other required security screenings.
• Experience in data science, machine learning, experimentation, and AI with a track record of delivering results (preferred).
• Experience in program management and leading cross-functional teams (preferred).
• Experience with LLM finetuning, reinforcement learning, evaluation techniques, RAG, and agentic workflows (preferred).
• Development experience in Python (preferred).
• Analytical, problem-solving, and presentation skills (preferred).