The Databricks Solution Architect joins the IBM Consulting team to lead the end-to-end architecture for a large-scale data modernization program. You will define target data and AI architectures across AWS and GCP, managing complex data migrations, modeling standards, and lakehouse designs. Daily responsibilities include establishing engineering patterns for ingestion, transformation, and orchestration while designing ML/GenAI lifecycles including RAG and MLOps. You will utilize technologies such as Spark, PySpark, Scala, Delta Lake, MLflow, Unity Catalog, and Apache Iceberg. The role requires expertise in integrating SAP-sourced data, managing cross-cloud movements, and implementing security guardrails for responsible AI. You will solve technical challenges by translating business objectives into scalable, policy-compliant designs that integrate AI capabilities into enterprise workflows across diverse hyperscaler environments.
Define target data and AI architectures across AWS/Databricks and GCP-based ML/GenAI services.
Lead data migration architecture including re-engineering, ingestion patterns, sequencing, and risk management.
Own data modeling standards for conceptual, logical, and physical models across lakehouse layers.
Define data engineering patterns for ingestion, transformation, orchestration, CI/CD, and cost control.
Architect the reporting layer including semantic models, governed metrics, and self-service access.
Design the ML/GenAI lifecycle including feature engineering, model serving, RAG, and MLOps.
Architect secure cross-cloud data movement and integration across AWS, Azure, and GCP.
Define security, data privacy, and responsible AI governance guardrails for all AI use cases.
What we're looking for
Must possess a Bachelor's Degree.
Must be able to travel 4 days per week.
Must be a Databricks Certified Data Engineer or Machine Learning Professional.
Must have proven expertise architecting enterprise-scale data, AI/ML, and Generative AI solutions.
Must have advanced expertise in data/solution architecture, enterprise data migration, data modeling, and cloud-based data engineering using Databricks and AWS.
Must have hands-on experience with Spark (PySpark/Scala), Delta Lake, MLflow, Unity Catalog, and large-scale batch and streaming workloads.
Must have experience architecting enterprise AI/GenAI solutions including RAG, agents, MLOps/LLMOps, and observability across AWS, Azure, and GCP.
Must have strong expertise in integration, security, and architecture leadership including APIs, microservices, event-driven patterns, and Kafka.
Generative AI
LLMs
RAG
Agentic AI
LangChain
LangGraph
Semantic Kernel
AutoGen
CrewAI
Azure
Azure Databricks
Microsoft Fabric
Azure OpenAI
Azure AI Foundry
Azure Machine Learning
AKS
Delta Lake
Vector Databases
Knowledge Graphs
Azure Synapse Analytics
Azure Data Lake Storage
Cosmos DB
Azure SQL
Azure AI Search
Event Hubs
Service Bus
API Management
Microservices
Data Engineering