Senior Software Engineer, Spark Platform

What you'll do

• Set the multi-year technical direction for the in-house Spark-on-Kubernetes platform.
• Solve complex distributed systems problems regarding shuffle architecture, multi-tenant scheduling, and runtime performance.
• Partner with the Engineering Manager on the technical roadmap, hiring, and team structure.
• Mentor the engineering team through design reviews and by raising the bar on shipped code.
• Represent the Spark Platform in cross-team architecture forums with Data Engineering and ML Platform teams.
• Manage the reliability and observability of Spark cluster deployments at scale.

What we're looking for

• B.S., M.S., or PhD in Computer Science or equivalent.
• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep, hands-on experience with Apache Spark internals, query execution, and shuffle at platform scale.
• Production experience with remote/external shuffle systems, batch/big-data schedulers, or distributed compute observability patterns.
• Strong fluency operating workloads on Kubernetes in production, including operator patterns and network topology.
• Familiarity with data lake table formats such as Apache Iceberg or Delta Lake.
• Track record of acting as a technical leader on a platform team.
• Professional experience with Scala, Java, Python, or Go and strong SQL.