onsite
Data Engineer - Searchability NS&D
Data Engineer
Design, build, and maintain robust data pipelines and integration solutions using Apache technologies, SQL and NoSQL databases, supporting mission‑critical analytics for national security operations.
About the role
Key Responsibilities
- Develop, test, and optimise end‑to‑end data pipelines for ingesting, transforming, and loading large‑scale datasets.
- Implement ETL processes that ensure data quality, consistency, and timely delivery to downstream analytics platforms.
- Integrate heterogeneous data sources, including relational (SQL) and non‑relational (NoSQL) stores, into a unified data architecture.
- Leverage Apache technologies (e.g., Spark, Flink) to enable high‑performance batch and streaming workloads.
- Collaborate with data scientists, security engineers, and product teams to translate business requirements into scalable data solutions.
Requirements
- Proven experience designing and operating data pipelines and ETL workflows in a secure, high‑availability environment.
- Strong proficiency with Apache ecosystem tools (Spark, Hadoop, or similar) and scripting languages for data processing.
- Hands‑on experience with both SQL (e.g., PostgreSQL, Oracle) and NoSQL databases (e.g., MongoDB, Cassandra).
- Ability to work under an active Enhanced DV clearance and adhere to strict security protocols.
- Solid problem‑solving skills and the ability to communicate technical concepts to cross‑functional teams.