Senior AI & Data Engineer

What you'll do

• Architect, develop, and maintain scalable, low-latency data pipelines using Azure Databricks and Spark.
• Design and optimize Lakehouse architecture using Delta Lake tables following Bronze, Silver, and Gold layers.
• Design and implement autonomous agents using Databricks Mosaic AI Agent Framework, LangChain, or AutoGen.
• Build and optimize vector search indexes using Databricks Vector Search for accurate context retrieval.
• Process large-scale distributed datasets on the Databricks Intelligence Platform.
• Design and implement data models that meet business requirements and enhance data usability.
• Monitor pipeline health and performance by implementing automated alerting and observability frameworks.

What we're looking for

• Must have a minimum of three years of professional experience as a Data & AI Engineer or in a related role.
• Must have a minimum of three years of experience working with Microsoft Azure cloud data services.
• Must have a minimum of two years of experience with Databricks, including PySpark, SQL, Delta Lake, and workflows.
• Must possess a Bachelor's degree, or an Associate Degree with at least 6 years of work experience, or 12 years of work experience.
• Advanced Python (PySpark) and SQL for data transformation and pipeline development (preferred).
• Experience with Azure Databricks, Spark, Delta Lake, Auto Loader, and Lakehouse Medallion Architecture (preferred).
• Experience with Git-based version control, CI/CD pipelines, MLOps/LLMOps, and specific certifications like Databricks Certified Data Engineer Professional (preferred).