remote
Azure Data Engineer - Unisys
Data Engineer
Design, build, and automate end‑to‑end ETL/ELT pipelines on Azure and AWS, leveraging Azure Data Factory, AWS Glue, Airflow, PySpark, Scala, and real‑time streaming with Kafka and Delta Lake.
About the role
Key Responsibilities
- Architect, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory, AWS Glue, and Apache Airflow.
- Implement large‑scale data processing jobs with PySpark and Scala on Databricks or EMR clusters.
- Design and deploy real‑time ingestion and processing solutions using Apache Kafka, Confluent, or AWS Kinesis.
- Manage cloud data lake storage (ADLS Gen2, S3) and enable ACID‑compliant transactions with Delta Lake or Apache Iceberg.
- Create performant data models and schemas for cloud data warehouses, ensuring optimal query performance and data quality.
Requirements
- 5+ years of experience in data engineering, with strong hands‑on expertise in Azure Data Factory and AWS data services.
- Proficiency in PySpark and Scala for distributed data processing.
- Experience building streaming pipelines using Apache Kafka, Confluent, or AWS Kinesis.
- Solid understanding of data lakehouse concepts, Delta Lake or Apache Iceberg, and cloud storage optimization.
- Ability to translate business requirements into robust, maintainable data solutions.