onsite
Senior Data Engineer Contractor - SmartestEnergy
Data Engineer
Lead the design, development, and maintenance of scalable data pipelines using Python, Spark, and AWS services, ensuring high data quality and performance for enterprise analytics.
About the role
Key Responsibilities
- Architect and implement robust data pipelines from diverse sources into cloud data warehouses.
- Optimize ETL processes using Apache Spark and Python for performance and reliability.
- Collaborate with data scientists and analysts to deliver clean, well‑documented datasets.
- Monitor pipeline health, troubleshoot issues, and implement automated alerts.
- Maintain data governance, security, and compliance across all data assets.
Requirements
- 5+ years of experience in data engineering with a focus on large‑scale data processing.
- Proficiency in Python, SQL, and Spark (PySpark).
- Hands‑on experience with AWS services such as S3, Redshift, Glue, and Lambda.
- Strong understanding of ETL best practices, data modeling, and performance tuning.
- Excellent problem‑solving skills and ability to work independently in a contractor role.
Skills
pythonsqlapache sparkaws