As a Senior Data Engineer at our firm in New York, you will join the AI & Analytics Infrastructure team to design and scale data pipelines for a critical Agentic AI engagement. Your day-to-day responsibilities include developing robust data pipelines using Microsoft Fabric, Databricks, or Azure Synapse Analytics, ensuring they are performant and reliable for enterprise-scale operations. You will also architect and maintain data infrastructure, implement data lakehouse patterns, and collaborate with AI engineers to ensure structured data outputs. Additionally, you will establish data governance standards, enforce validation frameworks, and optimize storage environments for cost efficiency. Ideal candidates have extensive experience in Azure Data Factory, AWS Glue, Databricks, and Snowflake, along with a background in MLOps and feature engineering for AI models. This role demands expertise in cloud-native deployment practices and secure enterprise data operations across Azure and AWS platforms.
Design, build, and maintain robust data pipelines using Microsoft Fabric, Databricks, and Azure Synapse Analytics.
Develop scalable architectures to ensure data pipelines are performant and reliable for enterprise use.
Define and implement data lakehouse patterns and medallion architecture for AI and analytics use cases.
Implement data quality checks and validation frameworks to ensure trustworthy data outputs.
Establish and enforce data governance standards including lineage tracking, cataloging, and access controls.
What we're looking for
7+ years of experience designing and maintaining scalable batch and real-time data pipelines on Azure and AWS.
Expertise in building and optimizing enterprise data platforms using Azure Data Factory, Azure Data Lake, AWS S3, AWS Glue, Databricks, and Snowflake.
Development of robust ETL/ELT frameworks for analytics, reporting, operational, and AI/ML use cases across cloud and hybrid ecosystems.
Implementation of scalable ingestion and transformation pipelines for structured, semi-structured, and unstructured data sources.
Support for data industrialization through reusable pipeline frameworks, standardized engineering practices, observability, monitoring, automated testing, and CI/CD deployment patterns.
Enablement of trusted enterprise data foundations by implementing data quality controls, metadata management, lineage tracking, cataloging, and governance capabilities.
SQL
Python
Snowflake
Google BigQuery
AWS Redshift
Databricks
Delta Lake
dbt
Apache Airflow
Flyte
CI/CD
Terraform
Kubernetes
Tableau
Power BI
Looker
Apache Iceberg
Palantir Foundry
Rippling API
Workday API
Oracle HCM Cloud API