onsite
Senior Data Engineer - BasisPath
Data Engineer
Senior Data Engineer responsible for designing, building, and operating cloud‑native data pipelines and platforms that ingest, transform, and analyze large‑scale operational data for government clients.
About the role
Key Responsibilities
- Architect and maintain scalable, cloud‑based data platforms on AWS to support mission‑critical data acquisition and analytics.
- Design, develop, and optimize ETL/ELT pipelines using Python, SQL, Apache Spark, and Airflow for high‑volume, real‑time data streams.
- Implement robust data ingestion and streaming solutions with Apache Kafka and related technologies.
- Ensure data quality, governance, and security across the entire data lifecycle.
- Collaborate with cross‑functional teams to translate operational requirements into technical solutions and provide ongoing support.
Requirements
- 5+ years of professional experience building data pipelines and platforms in a cloud environment, preferably AWS.
- Strong programming skills in Python and advanced SQL for data transformation and analysis.
- Hands‑on experience with distributed processing frameworks such as Apache Spark and streaming platforms like Apache Kafka.
- Proficiency with workflow orchestration tools (e.g., Airflow) and containerization (Docker) for reproducible deployments.
- Solid understanding of data modeling, data warehousing, and best practices for data security and governance.
Skills
pythonsqlawsapache sparkairflowdocker