remote
Senior Data Engineer - Headstrong Services
Data Engineer
Senior Data Engineer responsible for designing, building, and scaling data pipelines and platforms using Python, Spark, Airflow, and cloud services to enable AI‑driven analytics and real‑time data processing.
About the role
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines on cloud platforms (AWS) using Python, Spark, and Airflow.
- Implement real‑time data ingestion and streaming solutions with Kafka and other messaging systems.
- Collaborate with data scientists and analytics teams to provide clean, well‑documented data sets for AI and machine‑learning models.
- Optimize data storage, query performance, and data models in relational and big‑data environments.
- Ensure data quality, governance, and security compliance across all pipelines.
Requirements
- 5+ years of hands‑on experience in data engineering, building large‑scale data pipelines.
- Proficiency in Python, SQL, and big‑data processing frameworks such as Apache Spark.
- Experience with workflow orchestration tools like Apache Airflow.
- Strong knowledge of cloud services (AWS) and data streaming technologies (Kafka).
- Solid understanding of data modeling, warehousing concepts, and performance tuning.
Skills
pythonsqlapache sparkawskafka