onsite
GCP Data Engineering - Lead Programmer Analyst - Bilvantis Technologies
Software Engineer
Lead Programmer Analyst with 5–8 years of data engineering experience, specializing in GCP services such as BigQuery, Dataflow, Pub/Sub, and Dataform. Expert in SQL, Python, PySpark, and orchestrating data pipelines with Airflow and CI/CD workflows.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines on GCP using BigQuery, Dataflow, DataProc, and Dataform.
- Implement robust ETL processes with PySpark and Python, ensuring data quality and performance.
- Orchestrate workflows and schedule jobs using Apache Airflow, integrating with CI/CD pipelines for automated deployments.
- Collaborate with cross‑functional teams to define data models, optimize queries, and troubleshoot production issues.
- Manage version control with Git, enforce code review standards, and document architecture decisions.
Requirements
- 5–8 years of professional data engineering experience.
- Deep expertise in GCP services: BigQuery, Dataflow, Pub/Sub, Dataform, DataProc, and Cloud SQL.
- Proficient in SQL, Python, and PySpark for data transformation and analytics.
- Hands‑on experience with Apache Airflow, CI/CD pipelines, and Git workflows.
- Strong problem‑solving skills, ability to work independently, and excellent communication.