Databricks Data Engineer, Lead

What you'll do

• Design, build, and maintain robust end-to-end data pipelines within Databricks and AWS environments.
• Develop and optimize ETL processes using Python, PySpark, Delta Lake, and Unity Catalog.
• Design data models using dimensional and relational structures within a medallion architecture.
• Implement performance optimization techniques such as partitioning, bucketing, and caching for ETL jobs.
• Oversee the assessment, design, and maintenance of scalable data platforms for clients.
• Provide technical leadership and mentorship to a multi-disciplinary team of engineers and analysts.
• Ensure data security, compliance, and governance standards are met across all data flows.
• Troubleshoot and resolve complex issues using comprehensive logging and error handling mechanisms.

What we're looking for

• 8+ years of experience designing, developing, deploying, and managing ETL processes and data pipelines with robust logging and error handling.
• 3+ years of experience building and optimizing data pipelines using Databricks notebooks, clusters, and tools like Delta Lake and Unity Catalog.
• 3+ years of experience writing Python and PySpark code for ETL scripts and custom transformations.
• Experience in data modeling using dimensional vs relational models within a medallion architecture.
• Experience using AWS services including AWS Glue, Amazon S3, and Amazon RDS.
• Knowledge of performance optimization techniques such as partitioning, bucketing, and caching for ETL jobs.
• Ability to obtain and maintain a Public Trust or Suitability/Fitness determination.
• Bachelor’s degree in Computer Science, Computer Engineering, Engineering, Mathematics, Physics, Information Systems, or Information Technology.