onsite
Senior IT Big Data Developer - Centurion Consulting Group
Software Engineer
Lead the design and optimization of large‑scale data pipelines for economic research, leveraging Big Data technologies, ETL processes, and cloud services to deliver high‑performance analytics solutions.
About the role
Key Responsibilities
- Architect, develop, and maintain scalable data pipelines using Hadoop, Spark, and related Big Data technologies.
- Design and implement efficient ETL processes to ingest, transform, and load large volumes of economic data.
- Collaborate with data scientists and analysts to ensure data quality, accessibility, and performance for research and policy analysis.
- Optimize data storage and retrieval strategies on AWS, including S3, Redshift, and EMR.
- Monitor, troubleshoot, and tune pipeline performance, ensuring reliability and minimal downtime.
Requirements
- 5+ years of experience in data architecture and engineering within a large, complex environment.
- Experience with data modeling, metadata management, and data governance best practices.
- Excellent problem‑solving skills and ability to communicate technical concepts to non‑technical stakeholders.
Skills
sqlpythonapache sparkaws