remote
Principal Engineer- Data Platform, TD Securities
Software Engineer
Lead the design and evolution of a high‑performance data platform, driving scalable data pipelines, cloud integration, and advanced analytics for enterprise‑grade solutions.
About the role
Key Responsibilities
- Architect and oversee end‑to‑end data pipelines using Spark, Hadoop, and SQL to support real‑time and batch processing at scale.
- Lead migration and optimization of on‑prem data services to AWS, ensuring high availability, security, and cost efficiency.
- Collaborate with data scientists and business stakeholders to translate analytical requirements into robust, production‑ready data solutions.
- Mentor and coach a team of data engineers, fostering best practices in coding, testing, and documentation.
- Drive continuous improvement of data platform performance, monitoring, and observability.
Requirements
- 10+ years of experience in data engineering or related roles, with a proven track record of leading large‑scale data platform projects.
- Deep expertise in Python, Apache Spark, Hadoop ecosystem, and SQL-based data warehouses.
- Hands‑on experience with AWS services (EMR, Redshift, S3, Glue, Lambda) and cloud architecture best practices.
- Strong analytical, problem‑solving, and communication skills, with the ability to translate complex technical concepts to non‑technical stakeholders.
- Experience with CI/CD pipelines, containerization (Docker/Kubernetes), and automated testing in data environments is a plus.
Skills
pythonapache sparkhadoopsqlaws