Senior AI Platform Engineer, Data and Systems

What you'll do

• Design and build streaming-first data pipelines using event-driven architectures to reduce end-to-end latency.
• Manage the ML Attribute Store by building low-latency online serving capabilities and unified batch/streaming aggregations.
• Develop MCP-compatible Agent Data APIs to make the lakehouse discoverable and queryable by autonomous AI agents.
• Build an agentic framework for automated anomaly detection, pipeline self-healing, and root cause analysis.
• Drive operational excellence through observability, incident response automation, performance tuning, and cost optimization.
• Translate platform capabilities into self-serve infrastructure to reduce engineering toil for non-platform teams.
• Integrate LLMs into production workflows using prompt engineering, tool-use, and structured output parsing.
• Implement automated data governance systems to replace manual operational tasks with self-healing pipelines.

What we're looking for

• Hold a BS/MS in Computer Science, Engineering, or equivalent practical experience.
• Have 6+ years of experience in data platform engineering, distributed systems, or backend infrastructure at scale.
• Possess deep hands-on experience with Apache Spark, Databricks, Delta Lake, or similar lakehouse technologies.
• Demonstrate a track record of building and operating large-scale pipelines processing billions of events daily.
• Have strong experience with streaming systems such as Kafka, Kinesis, Flink, or Spark Structured Streaming.
• Be proficient in Python and/or Scala, with SQL fluency required; Java or Go is a plus.
• Experience with cloud platforms (AWS or Azure), containerization (Docker, Kubernetes), and CI/CD for data pipelines.
• Possess production experience integrating LLMs into workflows using agentic AI frameworks like LangChain, LangGraph, or CrewAI.