onsite
Senior Lead Data Engineer - Capital One
Data Engineer
Lead data engineering initiatives by designing scalable pipelines, modernizing data platforms, and guiding Agile teams using Python, Spark, Kafka, and AWS to deliver robust, business‑focused solutions.
About the role
Key Responsibilities
- Architect and implement end‑to‑end data pipelines on cloud platforms, leveraging Python, Spark, and Kafka to ingest, transform, and store large‑scale datasets.
- Partner with product owners, analysts, and cross‑functional Agile teams to translate business requirements into technical designs and data models.
- Drive migration and modernization of legacy data systems to AWS services, ensuring high availability, security, and cost efficiency.
- Mentor and lead a team of data engineers, fostering best practices in code quality, testing, CI/CD, and documentation.
- Establish and enforce data governance, quality, and observability standards using monitoring tools and automated testing frameworks.
Requirements
- 5+ years of hands‑on experience building data pipelines with Python, SQL, and Apache Spark.
- Strong expertise in streaming technologies such as Kafka and cloud services (AWS S3, Redshift, Glue, EMR).
- Proven track record designing data models and ETL processes for enterprise‑scale analytics.
- Experience leading technical teams in an Agile environment, including code reviews, mentorship, and delivery planning.
- Solid understanding of data governance, security, and performance optimization in cloud‑native architectures.
Skills
pythonsqlapache sparkkafkaaws