We are seeking a Senior Databricks Data Engineer with extensive hands-on experience in developing high-throughput and configurable data pipelines using Databricks. This role involves designing, building, and testing end-to-end data ingestion pipelines for streaming, events, and batch processes, as well as deploying scalable ETL pipelines on cloud-based infrastructure using Databricks SQL and Scala/PySpark. The ideal candidate will optimize job performance through query optimization and resource management, troubleshoot DevOps pipelines and Azure services, and collaborate effectively with cross-functional teams to drive project success. Strong skills in microservices architecture, critical thinking, and agile development methodologies are essential. Experience in the financial industry and handling large datasets on platforms like Databricks and Azure is highly valued.
Design and develop custom high-throughput data pipelines using Databricks.
Build and test end-to-end data ingestion pipelines for streaming, events, and batch processing.
Develop scalable ETL pipelines on cloud-based infrastructure with Databricks SQL & Scala/PySpark.
Optimize and tune Databricks job performance through query optimization and resource management.
Address data issues by conducting impact analysis, root cause analysis, and implementing corrective actions.
What we're looking for
7+ years of overall IT experience with a focus on data engineering.
3+ years of hands-on Databricks experience using SQL & Scala/PySpark.
Expertise in designing and deploying scalable ETL pipelines on cloud-based infrastructure.
Strong critical thinking, communication, and problem-solving skills.
Experience working in the financial industry and collaborating across geographical locations.
Market check
Salary context
This
$90,000–$157,500
range sits above
11%
of similar postings on FindRole.
Peer median band
$125,760–$202,500
Median floor and ceiling across peers.
Typical midpoint (25–75%)
$126,800–$218,293
Middle half of comparable postings.
Based on 240 comparable postings.*
* 240 is the maximum number of comparable postings sampled.
Employer
About State Street
State Street Corporation is one of the world''s largest custodian banks and asset managers, providing investment servicing, investment management, and investment research to institutional investors. Industry: Financial Services & Asset Custody
State Street currently has
133 open roles
on FindRole.
Listed pay typically runs
$110,000–$180,000
across 131 roles with salary data.
Databricks
AWS
PySpark
Python
Delta Lake
CI/CD
SQL
Power Platform
APIs
Docker
Hadoop
S3
Glue
Lambda
IAM
KMS
Unity Catalog
Power BI
Power Apps
Terraform
PostgreSQL
Oracle
JSON
Parquet