onsite
Lead Data Engineer - Intelligent Foundations and Experiences - Capital One
Data Engineer
Lead a team of data engineers to design, build, and scale data pipelines and platforms using Python, Spark, and AWS, while integrating real‑time streaming with Kafka and delivering robust data models for business insights.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines and lakehouse solutions on AWS using Python, SQL, and Apache Spark.
- Lead cross‑functional Agile teams, providing technical guidance and fostering a collaborative, inclusive development environment.
- Implement real‑time data ingestion and processing architectures with Kafka and streaming services.
- Define and enforce data modeling standards, data quality frameworks, and governance practices.
- Drive performance optimization, cost management, and reliability of cloud‑based data platforms.
Requirements
- 5+ years of hands‑on experience building large‑scale data pipelines and platforms in a cloud environment (AWS preferred).
- Strong proficiency in Python, SQL, and Apache Spark for batch and streaming workloads.
- Experience with Kafka or similar event‑streaming technologies and data modeling best practices.
- Demonstrated ability to lead technical teams, mentor engineers, and work effectively in Agile settings.
- Solid understanding of cloud architecture, CI/CD pipelines, and infrastructure‑as‑code tools.
Skills
pythonsqlapache sparkawskafka