Principal Data Engineering Lead, Services Special Project

What you'll do

• Build and implement batch and streaming ETL/ELT pipelines for diverse data sources including real-time events and unstructured media.
• Develop and maintain Kafka-based ingestion and processing pipelines to ensure reliable data delivery into the data lake.
• Create robust logical and physical data models using dimensional modeling and optimized storage patterns like Parquet or ORC.
• Establish and enforce data quality checks, SLAs, and observability standards to ensure data accuracy and reliability.
• Integrate raw signals with metadata and attribution to power downstream analytics, billing, and optimization use cases.
• Implement governance frameworks for data lineage, schema management, versioning, and security in compliance with privacy standards.
• Build pipelines that process multimodal data and integrate ML model inference, including LLMs and embedding models.
• Optimize real-time inference performance by tuning batching, KV-cache, and GPU utilization within the data pipeline.

What we're looking for

• Master's degree required.
• 12+ years of experience in data engineering and building large-scale ETL/ELT pipelines.
• Proficiency in dimensional modeling, schema design, and managing SQL/NoSQL databases like Postgres, Cassandra, and Redis.
• Expertise in distributed processing frameworks including Apache Spark, BigTable, Hadoop, Kafka, Iceberg, and Flink.
• Strong software engineering fundamentals with experience in Scala, Java, Python, and PySpark.
• Experience with workflow orchestration (Airflow, Beam), containerization (Docker, Kubernetes), and CI/CD tools like Jenkins.
• Experience building pipelines for multimodal data and deploying/optimizing LLMs and ML models in production runtimes.
• Knowledge of data governance, security best practices, and experience collaborating with executive leadership.