onsite
GCP Senior Data Lead - HCLTech
Software Engineer
Senior GCP Data Lead responsible for designing, building, and optimizing large‑scale data pipelines on Google Cloud Platform using BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, and Python, with infrastructure automation via Terraform and AI integration through Vertex AI.
About the role
Key Responsibilities
- Architect and implement end‑to‑end data pipelines on GCP, leveraging BigQuery, Dataflow, Dataproc, and Pub/Sub to ingest, process, and store massive datasets.
- Design and maintain workflow orchestration using Cloud Composer, ensuring reliable scheduling and monitoring of data jobs.
- Develop reusable, production‑grade Python code for data transformation, validation, and enrichment.
- Automate infrastructure provisioning and CI/CD pipelines with Terraform and GitHub Actions, supporting a DevOps culture.
- Integrate Vertex AI services to embed machine‑learning models into data workflows.
- Collaborate with cross‑functional teams to define data requirements, optimize performance, and enforce best practices for security and cost efficiency.
Requirements
- 7+ years of experience as a GCP Data Engineer or similar role.
- Deep expertise with Google BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, and Python.
- Hands‑on experience automating cloud resources using Terraform and managing CI/CD pipelines with GitHub Actions.
- Familiarity with Vertex AI and data preparation tools such as DataPrep.
- Strong problem‑solving skills and ability to work independently in a fast‑paced environment.