onsite
Azure Data Factory & Spark Data Engineer - BridgingIT GmbH
Data Engineer
Lead the design, development, and maintenance of scalable data pipelines using Azure Data Factory and Apache Spark, ensuring high-quality data integration and processing for enterprise analytics.
About the role
Key Responsibilities
- Design, build, and optimize data pipelines in Azure Data Factory to ingest, transform, and load data from diverse sources into Azure Data Lake and SQL Data Warehouse.
- Develop and maintain Spark jobs (PySpark/Scala) for large-scale data processing, ensuring performance, reliability, and scalability.
- Collaborate with data scientists and business analysts to translate business requirements into robust data models and ETL solutions.
- Implement data quality checks, monitoring, and alerting to guarantee data integrity and availability.
- Document architecture, code, and best practices; mentor junior engineers on Azure and Spark technologies.
Requirements
- 3+ years of experience in data engineering with Azure Data Factory and Apache Spark.
- Strong proficiency in Python and SQL; experience with PySpark or Scala is a plus.
- Hands‑on knowledge of Azure Data Lake, Azure Synapse Analytics, and related services.
- Solid understanding of data modeling, ETL patterns, and performance tuning.
- Excellent problem‑solving skills and a collaborative mindset.
Skills
apache sparkpythonsql