onsite
Senior Data Engineer - Clarity Innovations, LLC
Data Engineer
Senior Data Engineer leading design and implementation of secure, scalable data pipelines and platforms using Python, SQL, Airflow, AWS, Docker, and CI/CD to transform raw data into actionable intelligence for national security missions.
About the role
Key Responsibilities
- Design, develop, and maintain robust data pipelines and ETL processes using Python, SQL, and Apache Airflow.
- Implement cloud-native solutions on AWS, leveraging services such as S3, Redshift, and Lambda for scalable storage and processing.
- Containerize applications with Docker and orchestrate deployments through CI/CD pipelines adhering to DevSecOps best practices.
- Integrate real‑time data streams using Kafka and build streaming analytics workflows.
- Collaborate with intelligence analysts and cyber operators to translate mission requirements into data architectures that support rapid decision‑making.
- Ensure data security, compliance, and governance across all stages of the data lifecycle.
Requirements
- 5+ years of experience in data engineering, with a strong focus on building production‑grade pipelines.
- Proficiency in Python, SQL, and workflow orchestration tools such as Apache Airflow.
- Hands‑on experience with AWS cloud services and containerization (Docker, Kubernetes).
- Demonstrated ability to implement CI/CD and DevSecOps practices for data platforms.
- Experience with real‑time messaging systems (Kafka) and big‑data processing frameworks (e.g., Spark) is a plus.
Skills
pythonsqlawsdockercicdkafka