remote
Data Engineer - Web Scraping - 10a Labs
Data Engineer
Lead the design and optimization of scalable web‑scraping pipelines on Google Cloud, transforming raw structured and unstructured data into actionable insights for AI safety and threat intelligence.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines for large‑scale web scraping using Python and GCP services such as Cloud Functions, Dataflow, and BigQuery.
- Implement robust data extraction, cleaning, and transformation logic to convert raw HTML, JSON, and other formats into structured datasets.
- Optimize pipeline performance and cost through efficient use of GCP resources, parallel processing, and incremental data loading.
- Collaborate with data scientists and threat‑intelligence analysts to define data schemas, quality metrics, and ingestion schedules.
- Monitor pipeline health, troubleshoot failures, and implement automated alerts and recovery mechanisms.
Requirements
- 3+ years of experience building production‑grade data pipelines in Python.
- Strong proficiency with Google Cloud Platform, including Dataflow, Cloud Functions, Pub/Sub, and BigQuery.
- Hands‑on experience with web scraping libraries (BeautifulSoup, Scrapy, Playwright) and handling dynamic content.
- Solid understanding of SQL and data modeling for analytical workloads.
- Excellent problem‑solving skills and ability to work independently in a fast‑paced environment.