Senior Data Engineer, Services Special Projects

What you'll do

• Build and maintain scalable batch and streaming ETL/ELT pipelines for diverse data sources including multimodal media and real-time events.
• Develop Kafka-based ingestion and processing pipelines to ensure reliable data delivery into the data lake.
• Create robust logical and physical data models optimized for reporting, storage, and operational use cases.
• Implement data quality checks, observability standards, and monitoring systems to ensure accuracy and reliability.
• Integrate raw signals with metadata and attribution to power downstream analytics, billing, and optimization tasks.
• Manage data governance including lineage, schema management, versioning, and security protocols.
• Build pipelines that process multimodal data and integrate ML model inference for data enrichment.
• Optimize production runtimes for LLMs and embedding models within high-throughput data pipelines.

What we're looking for

• 10+ years of experience in data engineering and building large-scale ETL/ELT pipelines.
• Proficiency in dimensional modeling, schema design, and managing big data lake architectures.
• Experience with distributed processing frameworks like Apache Spark, BigTable, Hadoop, and Flink.
• Strong software engineering fundamentals with proficiency in Scala, Java, Python, and PySpark.
• Hands-on experience with Apache Kafka, Iceberg, Airflow, Beam, and containerization tools like Docker and Kubernetes.
• Experience with cloud services (AWS), SQL/NoSQL databases, and analytics frameworks like Trino or Snowflake.
• Expertise in ML pipelines, including LLM inference optimization, multimodal data processing, and embedding storage.
• Masters Degree.