remote
Senior Data Engineer - Penn Mutual
Data Engineer
Senior Data Engineer designing and maintaining scalable batch and streaming pipelines on AWS, leveraging Python, Spark, SQL, Kafka, and Airflow to support enterprise analytics and reporting.
About the role
Key Responsibilities
- Design, build, and maintain scalable batch and streaming data pipelines that support enterprise analytics, reporting, and downstream consumption.
- Collaborate with architecture, analytics, data governance, and application teams to translate business and analytical needs into robust data engineering solutions.
- Implement data ingestion, transformation, and quality processes using Python, Apache Spark, and SQL on AWS services.
- Integrate streaming data sources with Kafka and orchestrate workflows with Airflow.
- Ensure data pipelines adhere to enterprise cloud and technology standards, including security, governance, and performance best practices.
Requirements
- 5+ years of experience in data engineering with a strong focus on batch and streaming pipelines.
- Hands‑on knowledge of workflow orchestration tools, preferably Airflow.
- Strong analytical skills, ability to troubleshoot complex data issues, and excellent communication with cross‑functional teams.
Skills
pythonapache sparksqlawskafkaairflow