Lead Software Engineer, Data Engineer

What you'll do

• Develop and debug high-quality production code using Python, PySpark, and Spark SQL.
• Design and execute data engineering solutions focusing on Spark-based ETL/ELT processes.
• Drive team adoption of enterprise-authorized AI-assisted engineering practices for code quality and delivery speed.
• Automate the remediation of recurring issues to improve data pipeline reliability and system stability.
• Lead evaluation sessions with vendors and internal teams to assess architectural designs and technical credentials.
• Lead communities of practice to promote best practices in Spark performance and Apache Iceberg operations.
• Manage and optimize Spark workloads on EMR or Databricks platforms for cost and performance.
• Implement CI/CD methods and automated testing for repeatable data pipeline deployments.

What we're looking for

• Formal training or certification on software engineering concepts and 5+ years of applied experience.
• 5+ years of applied experience building production data engineering and/or software engineering solutions.
• Hands-on experience delivering system design, application development, testing, and operational stability for large-scale data pipelines.
• Advanced proficiency in Python, PySpark, and Spark SQL with strong SQL fundamentals.
• Demonstrated experience leading the use of approved AI-assisted software development tools and understanding responsible AI use.
• Experience with AWS data management patterns, including S3 and AWS Glue Data Catalog.
• Required platform experience delivering and operating Spark workloads on EMR and/or Databricks.
• Required production experience with Apache Iceberg, including table design, partitioning, and metadata maintenance.
• Proficiency in automation and continuous delivery methods like CI/CD and automated testing.
• Kafka familiarity (preferred).
• Experience with Delta Lake concepts (preferred).
• Experience with Spark Structured Streaming and streaming ETL patterns (preferred).
• Working knowledge of Java (preferred).
• Experience using specific AI-assisted engineering tools like GitHub Copilot or Claude (preferred).