Staff Data Engineer

What you'll do

• Define and own the technical architecture and roadmap for enterprise data management and information infrastructure.
• Design systems and pipelines to ingest, cleanse, and normalize diverse internal and external datasets for AI use.
• Translate ambiguous business problems into prioritized technical strategies in partnership with senior IT and product leaders.
• Drive the delivery of data and API solutions that provide non-technical staff with natural-language access to governed data.
• Architect AI-ready pipelines for RAG workflows, including document parsing, chunking, embeddings, and vector search.
• Own the strategy and architecture of the governed semantic layer in Databricks Unity Catalog.
• Set enterprise standards for data quality, lineage, observability, and security access-control patterns.
• Provide technical leadership by mentoring engineers, conducting code reviews, and setting engineering standards.

What we're looking for

• Must have 10+ years of experience in data engineering or closely related roles, including senior-level experience.
• Must have extensive, hands-on Databricks experience with Delta Lake, Unity Catalog, Spark SQL, PySpark, and Workflows.
• Must possess expert-level SQL and Python skills.
• Must have a proven track record of designing and delivering a Databricks semantic layer adopted across multiple teams or domains.
• Must have experience architecting batch and streaming pipelines and lakehouse/medallion data models at enterprise scale.
• Must have experience designing and governing REST APIs with FastAPI or Flask, OAuth-based authentication, and Azure APIM.
• Must have strong working knowledge of AI data engineering, including embeddings, vector search, RAG, LLM agents, and AI evaluation.
• Must have deep experience with data governance, lineage, quality, and security in a large enterprise environment.
• Must have experience with cloud platforms (Azure preferred), CI/CD, and infrastructure practices.
• Must demonstrate technical leadership by mentoring engineers, leading design reviews, and influencing architecture across teams.
• Must have strong written and verbal communication skills to explain complex trade-offs to technical and non-technical stakeholders.
• Experience with Genie or similar text-to-SQL tools (preferred).
• Experience with MCP or agent frameworks for enterprise data access (preferred).
• Experience leading work across external vendors or delivery partners (preferred).
• Experience in financial data domains or finance data-sharing models (preferred).