Data Scientist, AI/ML Model Quality

What you'll do

• Curate and maintain gold-standard ground-truth datasets for ML and GenAI model evaluation.
• Audit training data for systemic bias and establish automated checks to detect data drift.
• Define and report key data quality metrics such as completeness, accuracy, and timeliness.
• Design automated data quality rules and integrate them into CI/CD workflows with engineering teams.
• Define and track ML observability metrics including performance, distribution, and training-serving skew.
• Develop real-time dashboards to provide stakeholders with a consistent view of model health.
• Analyze telemetry across GenAI workflows to monitor output coherence, latency, and regression patterns.
• Identify failure modes in GenAI systems and translate findings into actionable recommendations for product teams.

What we're looking for

• A Bachelor's degree is required, while a Master's or Ph.D. in a quantitative field is strongly preferred.
• 3+ years of experience in data science or a related role focusing on data quality, model evaluation, or ML observability.
• Proficiency in Python (Pandas, NumPy, Scikit-learn) and SQL for complex data analysis and metric creation.
• Experience querying and analyzing large-scale datasets using distributed computing frameworks like PySpark or Spark.
• Solid understanding of statistical methods including hypothesis testing, distribution analysis, and data drift detection.
• Experience defining and tracking ML model health metrics and observability in production environments.
• Familiarity with GenAI or LLM systems, including common quality failure modes and telemetry instrumentation.
• Experience with data visualization tools, LLM evaluation frameworks, or working with privacy-constrained data (preferred).