onsite
Data Engineer - Searchability Ltd
Data Engineer
Design, build, and maintain robust data pipelines and integration solutions using Apache technologies, SQL/NoSQL databases, and ETL processes for mission‑critical national security applications.
About the role
Key Responsibilities
- Develop, test, and deploy scalable data pipelines to ingest, transform, and load large volumes of structured and unstructured data.
- Design and implement ETL workflows that ensure data quality, consistency, and timely delivery to downstream analytics platforms.
- Integrate heterogeneous data sources—including relational, NoSQL, and streaming systems—into a unified data lake or warehouse.
- Leverage Apache technologies (e.g., Spark, Flink) to process data at scale and support real‑time analytics.
- Collaborate with data scientists, security engineers, and product teams to translate business requirements into technical solutions.
Requirements
- Proven experience building and maintaining data pipelines and ETL processes in a production environment.
- Strong knowledge of SQL and NoSQL databases, data modeling, and performance tuning.
- Hands‑on expertise with Apache ecosystem tools such as Spark or Flink.
- Ability to work with sensitive data and hold an active Enhanced DV Clearance (West).
- Excellent problem‑solving skills and ability to work both independently and within a multidisciplinary team.