onsite
Senior Specialist Solutions Architect - Data Engineering & Warehousing
Solutions Architect
Lead end‑to‑end data engineering solutions, architecting scalable pipelines on AWS and Azure, leveraging Apache Spark for big‑data processing and implementing anomaly detection and change data capture to ensure data integrity and real‑time insights.
About the role
Key Responsibilities
- Design, build, and maintain large‑scale data pipelines using AWS services (Glue, Redshift, S3) and Azure Data Factory.
- Implement Apache Spark jobs for batch and streaming data processing, optimizing performance and cost.
- Develop and deploy anomaly detection models to identify data quality issues in real time.
- Configure Change Data Capture (CDC) mechanisms across heterogeneous databases to enable incremental data loading.
- Collaborate with data scientists, BI teams, and stakeholders to translate business requirements into technical solutions.
- Ensure data governance, security, and compliance across cloud environments.
Requirements
- 5+ years of experience in data engineering and solutions architecture.
- Proficiency with AWS and Azure cloud platforms, including data services.
- Strong background in Apache Spark, Scala/Java/Python, and big‑data processing.
- Hands‑on experience with anomaly detection techniques and CDC implementations.
- Excellent communication skills and ability to work cross‑functionally.
Skills
awsapache sparkazure