| Microsoft Careers

What you'll do

• Design and implement large-scale data pipelines using PySpark and distributed processing frameworks.
• Build and maintain accurate data models representing system behavior and business logic.
• Develop validation mechanisms to ensure high standards of data correctness and consistency.
• Optimize data pipelines for performance, scalability, and cost efficiency.
• Define and enforce best practices for schema design and pipeline reliability.
• Collaborate with data scientists to support experimentation and analytics use cases.
• Ensure efficient data ingestion, processing, and storage in collaboration with platform engineers.

What we're looking for

• 5+ years of experience in data engineering with a focus on large-scale systems.
• Strong expertise in PySpark and distributed processing frameworks like Apache Spark.
• Proven track record of building and maintaining scalable data pipelines.
• Deep understanding of data modeling, schema design, and ensuring data quality.
• Proficiency in Python and familiarity with cloud-based data platforms (Azure, AWS).
• Experience with orchestration tools such as Airflow for managing workflows.