Spark Data Engineer, Senior

What you'll do

• Develop and deploy data pipelines to process petabyte-scale data from disparate sources.
• Design, build, and maintain scalable big data platforms for enterprise-level applications.
• Implement Spark development using PySpark or Java Spark to organize complex data structures.
• Create scripts and programs to convert various data types into usable formats.
• Model complex data structures across varied formats for analytical exploration.
• Monitor and operate data platforms to ensure consistent performance and scalability.
• Manage the assessment and design of distributed data and computing tools like AWS or Kafka.

What we're looking for

• 5+ years of experience in Spark development, including PySpark or Java Spark.
• 5+ years of experience designing, developing, operationalizing, and maintaining complex data applications at enterprise scale.
• 5+ years of experience with big data and building ETL pipelines to efficiently process petabyte scale data.
• Experience with modeling complex data structures of varied formats.
• Experience with distributed data and computing tools, including Spark, AWS, or Kafka.
• Experience creating solutions within a collaborative, cross-functional team environment.
• Ability to develop scripts and programs for converting various types of data into usable formats and support project team to scale, monitor and operate data platforms.
• Bachelor’s degree.