Senior Data Engineer, Services Special Projects

What you'll do

• Build and maintain scalable batch and streaming ETL/ELT pipelines for diverse data sources including multimodal media and real-time events.
• Develop Kafka-based ingestion and processing pipelines to ensure reliable data delivery into the data lake.
• Create robust logical and physical data models using dimensional modeling and optimized storage patterns like Parquet or ORC.
• Implement data quality checks, observability standards, and automated monitoring for anomaly detection and debugging.
• Integrate raw signals with metadata and attribution to power downstream analytics, billing, and optimization use cases.
• Manage data governance including lineage, schema management, versioning, and security protocols.
• Build pipelines that process multimodal data and integrate ML model inference, such as LLMs and embedding models, for enrichment.
• Optimize production runtimes for high-throughput real-time inference by tuning batching and GPU utilization.

What we're looking for

• 10+ years of experience in data engineering, including building and maintaining large-scale ETL/ELT pipelines.
• Proficiency in data modeling, specifically dimensional modeling and designing schemas for analytics and reporting.
• Experience with SQL/NoSQL databases (Postgres, Cassandra, Redis) and distributed processing frameworks like Apache Spark.
• Expertise in parallel processing frameworks such as BigTable or Hadoop.
• Strong software engineering fundamentals with proficiency in Scala, Java, Python, and PySpark.
• Hands-on experience with Apache Kafka, Iceberg, Flink, Airflow, Beam, and containerization tools like Docker and Kubernetes.
• Experience building pipelines for multimodal data and deploying/optimizing LLMs or ML models in production runtimes.
• Masters Degree.