onsite
DataOS Data Engineer - Information Technology Senior Management Forum
Data Engineer
Data engineer focused on scalable data ingestion, transformation, and integration to power business‑driven data products using Python, SQL, Spark, Airflow, and AWS services.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines for ingestion, transformation, and integration across multiple data sources.
- Implement ETL processes using Python, SQL, and Apache Spark to ensure high‑quality, reliable data for analytics and reporting.
- Leverage AWS services (S3, Glue, Redshift, Lambda) to build and optimize data workflows and storage solutions.
- Collaborate with data scientists and product teams to translate business requirements into robust data models and solutions.
- Monitor pipeline performance, troubleshoot issues, and continuously improve data processing efficiency.
Requirements
- Strong experience with Python, SQL, and ETL tooling.
- Hands‑on knowledge of Apache Spark and distributed data processing.
- Proficiency with AWS data services (S3, Glue, Redshift, Lambda).
- Solid understanding of data modeling, schema design, and data quality best practices.
- Excellent problem‑solving skills and ability to work independently or as part of a cross‑functional team.
Skills
pythonsqlawsapache sparkairflow