Data Engineer and Technical Lead

What you'll do

• Oversee the development and deployment of data pipelines and platforms to organize disparate data sources.
• Lead and mentor a multi-disciplinary team of analysts, data engineers, and developers in an agile environment.
• Design and maintain scalable data platforms using Databricks, Delta Lake, and medallion lakehouse architectures.
• Perform PySpark performance tuning on large datasets and manage Databricks Workflows and Jobs.
• Implement Unity Catalog information architecture, including catalog design, schema design, and governance.
• Apply software engineering practices such as Git, automated testing, and CI/CD for data assets.
• Develop dimensional data models, including star schemas and slowly changing dimensions, for Business Intelligence consumption.
• Manage AWS services including S3, IAM roles, and Secrets Manager to support Databricks infrastructure.

What we're looking for

• 8+ years of experience in data engineering or software engineering with a data focus, including Databricks in production.
• 7+ years of experience using Python and SQL, with demonstrated PySpark performance tuning on large datasets.
• 5+ years of experience with production including Delta Lake and bronze, silver, and gold medallion Lakehouse design.
• 3+ years of experience with Databricks Workflows, Jobs, Asset Bundles, and YAML-defined deployments.
• 3+ years of experience with Unity Catalog information architecture, governance, and software engineering practices like Git and CI/CD.
• Experience supervising engineers, leading projects, and Agile delivery within a collaborative, cross-functional team environment.
• Knowledge of AWS services such as S3, IAM roles, and Secrets Manager.
• Bachelor’s degree and Public Trust clearance.