remote
Senior Data Engineer - Lightfoot
Data Engineer
Senior Data Engineer building and operating end‑to‑end data platforms on AWS, designing scalable Spark pipelines, and ensuring robust data governance to unlock fleet efficiency and reduce emissions.
About the role
Key Responsibilities
- Design, build, and maintain a scalable data lake on AWS, ensuring high availability, security, and compliance.
- Develop and optimize large‑scale ETL pipelines using Apache Spark and Python to ingest, transform, and enrich in‑vehicle telemetry.
- Implement data governance frameworks, including metadata management, data quality checks, and access controls.
- Collaborate with data scientists and product teams to translate business requirements into data solutions that drive fleet efficiency.
- Monitor pipeline performance, troubleshoot issues, and continuously improve data processing workflows.
Requirements
- 5+ years of experience in data engineering with a strong focus on cloud (AWS) and big data technologies.
- Proficiency in Python, SQL, and Spark for building robust data pipelines.
- Hands‑on experience with AWS services such as S3, Glue, Redshift, and Lake Formation.
- Solid understanding of data modeling, schema design, and data governance best practices.
- Excellent problem‑solving skills and a passion for turning complex data into actionable insights.
Skills
pythonsqlapache sparkaws