Principal Engineer, Data Retrieval

What you'll do

• Architect and scale a unified data acquisition platform for discovering, crawling, and ingesting high-quality training data.
• Design scalable frontier storage and queueing systems with high-throughput writes, deduplication, and quality-based scheduling.
• Build a continuous discovery loop to extract links and feed new candidates back into the crawl frontier.
• Develop a shared crawl execution layer to fetch data from dynamic websites reliably and politely at scale.
• Build extensible pipelines for open-web, licensed, and partner content including text, PDFs, and multimodal assets.
• Implement provenance and policy enforcement controls for legal, privacy, and data governance requirements.
• Own production readiness including capacity planning, observability, fault tolerance, and cost efficiency.
• Instrument the pipeline end-to-end to set service objectives for throughput, freshness, and reliability.

What we're looking for

• A BS, MS, or PhD in Computer Science, Computer Engineering, or a related technical field, or equivalent experience.
• 15+ years of relevant experience in software engineering, including designing and operating large-scale distributed systems.
• Deep experience with web crawling or closely related discovery, indexing, or ingestion infrastructure.
• Strong hands-on engineering and system design skills, including performance analysis and production debugging.
• Experience building batch or streaming pipelines with a focus on consistency, retries, duplicate processing, backpressure, and recovery.
• Experience delivering data platforms for research, ML, search, or other demanding technical users.
• Strong technical communication and leadership skills to influence engineering, product, research, and governance teams.
• Experience architecting web-wide crawl platforms or supplying foundation-model pretraining data (preferred).