onsite
Principal Data Engineer - Metropolitan Council
Data Engineer
Lead the design, development, and optimization of large‑scale data pipelines and platforms, leveraging Python, Spark, and AWS to deliver reliable, high‑performance data solutions for regional planning and services.
About the role
Key Responsibilities
- Architect, build, and maintain robust, scalable data pipelines and warehouses supporting regional transportation, wastewater, and housing analytics.
- Lead the adoption of cloud‑native services (AWS) and big‑data technologies (Spark, Kafka) to improve data ingestion, processing, and storage.
- Collaborate with data scientists, analysts, and business stakeholders to translate requirements into efficient data models and ETL workflows.
- Establish best practices for data quality, governance, and security across the organization.
- Mentor and guide junior engineers, fostering a culture of continuous improvement and technical excellence.
Requirements
- 10+ years of experience in data engineering, with a proven track record designing enterprise‑grade data platforms.
- Strong proficiency in Python, SQL, and big‑data frameworks such as Apache Spark.
- Hands‑on experience with AWS services (e.g., S3, Redshift, Glue, Lambda) and streaming technologies like Kafka.
- Deep understanding of data modeling, ETL design, and performance tuning for large datasets.
- Excellent problem‑solving skills and ability to work cross‑functionally in a public‑sector environment.
Skills
pythonsqlapache sparkawskafka