Principal Data Engineering Lead, Services Special Project

What you'll do

• Build and implement batch and streaming ETL/ELT pipelines for diverse data sources including unstructured media and real-time streams.
• Develop and maintain Kafka-based ingestion and processing pipelines to ensure reliable data delivery into the data lake.
• Build robust logical and physical data models optimized for dimensional modeling, versioning, and storage patterns like Parquet or ORC.
• Define and enforce data quality checks, SLAs, and observability standards to ensure data accuracy and trustworthiness.
• Integrate raw signals with metadata and attribution to power downstream analytics, billing, and optimization use cases.
• Implement methodologies for data lineage, metadata management, schema governance, and security in alignment with privacy standards.
• Deliver solutions featuring logging, anomaly detection, and data validation with a focus on monitoring and debuggability.
• Advance the team's data stack by establishing standards for development, testing, deployment, and operations.

What we're looking for

• 12+ years of experience in data engineering, including building and maintaining large-scale ETL/ELT data pipelines.
• Proficiency in data modeling, dimensional modeling, and designing schemas optimized for analytics and reporting.
• Experience with SQL/NoSQL databases (including Postgres, Cassandra, Redis), Apache Spark, and parallel processing frameworks like BigTable/Hadoop.
• Strong software engineering fundamentals with experience in Scala, Java, Python, and PySpark.
• Hands-on experience with Apache Kafka, Iceberg, Flink, Airflow, Beam, and big data lake architectures.
• Experience with AWS services, containerization (Docker, Kubernetes/EKS), CI/CD tooling, and analytics frameworks like Trino.
• Experience building pipelines for multimodal data and ML model inference, including LLMs, and experience with data versioning tools (preferred).
• Masters Degree.