remoteonsite
Senior/Lead Databricks & AWS Solution Architect - CGI
Solutions Architect
Lead architect for Databricks Lakehouse solutions on AWS, driving scalable data pipelines, governance, and security while delivering business value through advanced data engineering practices.
About the role
Key Responsibilities
- Design, build, and optimize end‑to‑end data pipelines on the Databricks Lakehouse platform, leveraging Delta Lake for ACID transactions and schema enforcement.
- Integrate Databricks workloads with the AWS ecosystem (S3, Glue, Redshift, Athena, IAM, KMS) to ensure secure, compliant data flows.
- Architect scalable, high‑performance Spark jobs in Python/Scala, applying best practices for performance tuning, resource management, and cost control.
- Implement data governance, lineage, and cataloging using Unity Catalog, ensuring data quality and regulatory compliance.
- Collaborate with data scientists, analysts, and business stakeholders to translate requirements into robust, reusable data products.
- Mentor and lead a small team of developers, providing technical guidance and code reviews.
Requirements
- 7+ years of experience in data engineering with a strong focus on Databricks and AWS.
- Proven expertise in Delta Lake, Spark, and Python/Scala development.
- Deep knowledge of AWS services (S3, Glue, Redshift, Athena, IAM, KMS) and their integration with Databricks.
- Experience with data governance frameworks, Unity Catalog, and security best practices.
- Excellent communication skills and ability to translate complex technical concepts to non‑technical stakeholders.
Skills
databricksawsapache sparkpython