onsite
Data Engineer - NewGen Technologies
Data Engineer
Data Engineer responsible for designing, building, and maintaining scalable data pipelines and analytics infrastructure using Python, SQL, AWS, and Spark to support advanced cyber and intelligence operations.
About the role
Key Responsibilities
- Design, develop, and optimize large-scale data pipelines for ingestion, transformation, and storage using Python, SQL, and Spark.
- Implement and maintain data lakes and warehouses on AWS (S3, Redshift, Athena) ensuring high availability and performance.
- Collaborate with data scientists and analysts to provide clean, reliable datasets for machine learning and intelligence workflows.
- Monitor pipeline health, troubleshoot issues, and implement automated alerts and recovery procedures.
- Document data models, pipeline architecture, and best practices for future maintenance and scalability.
Requirements
- 3+ years of experience in data engineering with a strong focus on cloud-based solutions.
- Proficiency in Python, SQL, and Spark for data processing and transformation.
- Hands‑on experience with AWS services such as S3, Redshift, Glue, and Athena.
- Solid understanding of ETL concepts, data modeling, and performance tuning.
- Excellent problem‑solving skills and ability to work in a fast‑paced, mission‑critical environment.