Senior Manager, Machine Learning (Data Operations)

What you'll do

• Define annotation guidelines, taxonomies, and edge-case protocols for all labeling programs.
• Establish gold standard datasets and inter-annotator agreement targets to ensure data quality.
• Manage the Label Studio platform by defining project configurations and integration requirements.
• Translate machine learning model requirements into structured, unambiguous labeling tasks.
• Source, onboard, and manage external vendors to meet accuracy and throughput targets.
• Track operational metrics like cost per label and turnaround time to drive continuous improvement.
• Identify opportunities for active learning and pre-annotation to reduce costs without sacrificing quality.

What we're looking for

• 5+ years of experience in ML data operations, data labeling, or a related field like ML engineering or data science.
• Deep understanding of annotation quality frameworks including IAA, consensus labeling, gold standard evaluation, and calibration workflows.
• Direct experience managing labeling platforms such as Label Studio, Scale AI, Labelbox, or Prodigy.
• Proven track record of managing outsourced labeling vendors or BPOs for machine learning data production.
• Familiarity with common ML labeling tasks like text classification, NER, document extraction, and intent detection.
• Proficiency in Python and SQL to support labeling workflows and quality measurement.