Consultant, Data Engineering

What you'll do

• Process landed data through the medallion architecture to ensure structured sources meet freshness SLAs.
• Own the entity resolution process to consolidate multiple sources into a single canonical identity.
• Build model-based extraction lanes using NLP and LLMs to extract structure from unstructured content.
• Tag provenance and trust tiers on every record while quarantining failed data with specific reasons.
• Maintain high-volume data in the lakehouse and served layers to ensure performance and speed.
• Develop new approaches to generate business insights and specifications for reports based on business needs.
• Define project scopes and identify functional and technical requirements to produce accurate business insights.

What we're looking for

• Strong model expertise for unstructured data, including NLP and LLM-based extraction, document understanding, and embeddings.
• Strong PySpark and lakehouse/medallion experience in production.
• Experience with entity resolution and record linkage at scale.
• Proficiency with Airflow or comparable orchestration tools.
• Ability to manage data lineage, provenance, and trust tiers for all records.
• Ability to build model-based extraction lanes for unstructured content like HTML.
• Ability to maintain high-volume data in lakehouse and served layers for speed and lean performance.