Big Data Engineer Lead

S&P Global

Confirmed live yesterday High trust

Quick summary

Work type
On-site
Location
New York, NY
Salary
$149,302–$202,267 / yr
Posted
2 days ago
Freshness
Confirmed live yesterday
Closes
Sep 9, 2027

Market check

Salary context

Below market

How this pay compares to similar roles

Similar $195k
This role $176k
$140k most similar roles pay here $234k

This role pays less than 77% of similar roles. Most pay $177,900–$211,200 — the shaded band above. At the midpoint, this role pays about $176k versus about $195k for comparable roles.

Based on 240 similar postings.

Employer

About S&P Global

S&P Global delivers Essential Intelligence® that shapes decision making. We provide the world’s leading organizations with the right data, connected technologies and expertise they need to move ahead.

S&P Global currently has 46 open roles on FindRole.

Listed pay typically runs $142,000–$200,000 across 37 roles with salary data.

Most-posted roles

View all roles at S&P Global

At a glance

TL;DR · Big Data Engineer Lead

Big Data Engineer Lead will join the Ratings Organization's Data Services Team to provide technical leadership and develop scalable solutions for the next generation of analytics platforms. The role involves partnering with stakeholders to define use cases, collaborating with data scientists and software engineers to build machine learning and generative AI solutions, and developing tools to optimize model performance. Key responsibilities include analyzing large datasets, researching emerging AI technologies, and establishing technical standards across the organization. The required technical stack includes Python, Java, Spring Boot, SQL, and experience with frameworks like TensorFlow, PyTorch, or scikit-learn. Candidates should be proficient in cloud environments like AWS, containerization tools such as Docker and Kubernetes, and version control via Git. Specialized knowledge in LangChain, graph technologies, and distributed computing frameworks like Apache Spark is also preferred to solve complex data problems.

What you'll do

  • Gather requirements and define use cases from business stakeholders to plan solution delivery.
  • Design, build, and deploy scalable machine learning and generative AI solutions with engineering teams.
  • Analyze large datasets to develop data-driven insights that improve business outcomes.
  • Build platforms, services, and tooling to optimize and fine-tune machine learning models for performance.
  • Research and implement emerging AI technologies to enhance existing systems and drive innovation.
  • Establish technical standards and reusable AI capabilities across the organization.

What we're looking for

  • Experience building AI, machine learning, or data-driven solutions in cloud environments like AWS.
  • Strong Python programming skills using frameworks such as TensorFlow, PyTorch, or scikit-learn.
  • Hands-on experience developing applications with Java and Spring Boot.
  • Experience working with containerized and virtualized environments including Docker, Kubernetes, and virtual machines.
  • Familiarity with modern AI-assisted development tools like Cursor, Claude Code, or GitHub Copilot.
  • Proficiency in version control systems such as Git.
  • Strong knowledge of SQL and experience with structured and unstructured datasets.
  • Understanding of machine learning concepts including deep learning, NLP, and generative AI.
  • Experience building or working with AI agents, autonomous workflows, or agentic AI frameworks.
  • Experience with AI frameworks like LangChain, LangGraph, CrewAI, or similar technologies (preferred).
  • Experience with graph technologies and graph-based algorithms (preferred).
  • Familiarity with front-end technologies such as React, JavaScript, or HTML (preferred).
  • Experience in data engineering, data preparation, and feature engineering (preferred).
  • Experience working with data lakes or large-scale data ecosystems (preferred).
  • Experience building machine learning solutions using distributed computing frameworks like Apache Spark or Hadoop (preferred).
  • Knowledge of modern AI-native software development lifecycle practices (preferred).

More like this

Similar roles

Lead Software Engineer, Big Data

JPMorgan Chase

Plano, TX 44 days ago
Apache Spark Python Java Scala Spring Boot Docker Azure AWS Hadoop Iceberg Prometheus Grafana Elasticsearch CloudWatch CI/CD Git GitHub Bitbucket Databricks ETL
5+ yrs exp

Lead Data Science Engineer

Booz Allen Hamilton

McLean, VA 78 days ago $99,000$225,000
Python SQL ETL Data Pipelines Kubernetes Spark ELK OpenSearch AWS S3 Terraform Containerization Agile

Lead Data Engineer

Capital One Financial

McLean, VA 57 days ago $197,300$225,100
Python SQL Scala Java AWS Redshift Snowflake Spark Kafka Hadoop Hive EMR NoSQL Cassandra MySQL Linux Shell Scripting Agile
4+ yrs exp

Lead Data Engineer

Capital One Financial

Plano, TX 88 days ago $179,400$204,700
Python Java TypeScript React Databricks Snowflake Spark Kafka AWS Redshift SQL Scala Flink DynamoDB OpenSearch UNIX Linux CI/CD Agile Microservices
4+ yrs exp

Lead Data Engineer

Capital One Financial

McLean, VA +3 85 days ago $179,400$204,700
Python SQL Scala Java AWS Azure Google Cloud Redshift Snowflake Spark Kafka Hadoop Hive EMR NoSQL Cassandra MySQL UNIX Linux Agile
4+ yrs exp

Lead Data Engineer

Capital One Financial

McLean, VA +3 81 days ago $179,400$204,700
Python SQL Scala Java AWS Redshift Snowflake Spark Kafka Hadoop Hive EMR NoSQL Cassandra MySQL UNIX Linux Agile
4+ yrs exp