AI Data Engineer

What you'll do

• Build data pipelines to ingest, transform, and serve governed, AI-ready content from raw to gold layers.
• Implement medallion architecture patterns using Delta Lake tables with partitioning and optimization.
• Own the workflow-orchestration framework using Databricks Workflows and Delta Live Tables.
• Execute governance patterns including Unity Catalog structure, sensitivity classification, and access control.
• Build and operate retrieval-supporting infrastructure such as embedding pipelines and vector store maintenance.
• Define source-system contracts with Operations Capabilities regarding schemas, cadence, and quality thresholds.
• Design and operate data quality checks, freshness monitoring, and drift detection alerts.
• Contribute to and evolve a shared library of reusable pipeline patterns and code templates.

What we're looking for

• Must have a bachelor’s degree or equivalent.
• Must have at least four years of hands-on experience designing, building, and operating production data pipelines.
• Must have experience building data foundations for AI use cases, including embedding pipelines, vector stores, and retrieval evaluation.
• Must possess depth in Databricks and Spark, including Delta Lake, Delta Live Tables, Workflows, and Unity Catalog.
• Must be fluent in Python and SQL, with experience in PySpark, ETL patterns, Git, CI/CD, and Terraform.
• Must have experience with workflow orchestration tools like Databricks Workflows or Airflow.
• Must have foundational AWS knowledge, including IAM, S3, and KMS.
• Must have experience with knowledge graphs, entity resolution, or semantic data models (preferred).