onsite
DataOS Data Engineer - HP
Data Engineer
Data engineer focused on scalable data ingestion, transformation, and integration for business‑driven data products using Python, SQL, ETL pipelines, and cloud services like AWS and Spark.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data ingestion pipelines that transform raw data into usable formats for analytics and reporting.
- Implement robust ETL processes using Python, SQL, and Apache Spark to ensure data quality and consistency across multiple data sources.
- Collaborate with data scientists and product teams to build data models and data building blocks that support new data products and business initiatives.
- Leverage AWS services (S3, Glue, Redshift, Lambda) to orchestrate data workflows and optimize performance.
- Act as a technical coach, guiding team members on best practices, troubleshooting, and decision‑making for complex data challenges.
Requirements
- Strong experience with Python, SQL, and ETL tooling in a production environment.
- Hands‑on knowledge of AWS data services and experience building data pipelines on the cloud.
- Proficiency with Apache Spark or similar distributed processing frameworks.
- Solid understanding of data modeling, schema design, and data governance principles.
- Excellent problem‑solving skills and ability to work independently on complex, cross‑functional projects.
Skills
pythonsqlawsapache spark