onsite
Big Data/Spark Engineer - Software Guidance & Assistance
Software Engineer
Senior engineer to architect and optimize large‑scale data pipelines in a high‑volume AWS environment, leveraging Spark, SQL, and Python to deliver AI‑assisted workflows and automated validation at a regulatory client.
About the role
Key Responsibilities
- Design, develop, and maintain scalable Spark jobs to process terabytes of data in an AWS environment.
- Write efficient SQL queries and Python scripts for data extraction, transformation, and loading (ETL) pipelines.
- Implement AI‑assisted workflows and automated validation frameworks to ensure data quality and compliance.
- Collaborate with data scientists, DevOps, and regulatory teams to translate business requirements into technical solutions.
- Monitor, troubleshoot, and optimize performance of Spark clusters and related infrastructure.
Requirements
- 5+ years of experience in big data engineering with Spark, SQL, and Python.
- Proven expertise in AWS services (EMR, S3, Glue, Lambda, etc.) and large‑scale data processing.
- Strong knowledge of automation, CI/CD pipelines, and data validation techniques.
- Excellent problem‑solving skills and ability to work in a fast‑paced regulatory environment.
- Effective communication skills and a collaborative mindset.