remote
Data Engineer - Absolute Business Solutions Corp. (ABSc)
Data Engineer
Data Engineer with TS/SCI clearance building scalable data pipelines and architectures on AWS to support DIA’s OSINT knowledge platform, leveraging Python, SQL, and advanced ETL techniques.
About the role
Key Responsibilities
- Design, develop, and maintain robust data pipelines that ingest, transform, and store large volumes of structured and unstructured data for intelligence analysis.
- Implement scalable data architectures on AWS (S3, Redshift, Glue, Athena) ensuring high availability, performance, and security compliance.
- Collaborate with data scientists, analysts, and stakeholders to define data models, quality metrics, and reporting requirements.
- Optimize ETL processes using Python, SQL, and Spark, reducing processing time and improving data freshness.
- Monitor pipeline health, troubleshoot issues, and implement automated alerts and recovery mechanisms.
Requirements
- Minimum 3 years of experience in data engineering, with a strong focus on ETL and data pipeline development.
- TS/SCI clearance required; experience working with the Intelligence Community is a plus.