onsite
Data Engineer - MandelBulb Technologies
Data Engineer
We need a Data Engineer to design, build, and maintain scalable ETL/ELT pipelines, leveraging Apache Spark, Databricks, and Airflow on AWS for robust data integration and analytics.
About the role
Key Responsibilities
- Design, develop, and maintain high‑performance ETL/ELT pipelines for ingesting and transforming diverse data sources.
- Build and optimize large‑scale data processing jobs using Apache Spark and Databricks.
- Create, schedule, and monitor workflow orchestration with Apache Airflow.
- Integrate data from on‑premise systems, cloud services, and third‑party APIs into a unified data lake.
- Collaborate with analytics and BI teams to ensure data quality, reliability, and accessibility.
Requirements
- 3+ years of hands‑on experience building data pipelines in a cloud environment, preferably AWS.
- Proficiency with Apache Spark (PySpark or Scala) and Databricks for distributed processing.
- Strong knowledge of workflow orchestration using Apache Airflow.
- Solid understanding of ETL/ELT concepts, data modeling, and SQL.
- Experience with version control, CI/CD, and monitoring tools for data engineering workflows.
Skills
apache sparkdatabricksaws