remoteonsite
Lead Data Engineer - Aeries Technology Group
Data Engineer
Lead Data Engineer responsible for designing, building, and optimizing large-scale data pipelines and lakehouse architectures using Python, Spark, and AWS services to enable data-driven decision making across the organization.
About the role
Key Responsibilities
- Architect and develop end-to-end data pipelines from ingestion to analytics using Python, Spark, and AWS Glue.
- Design and maintain scalable data lakehouse solutions on S3 and Redshift, ensuring high availability and performance.
- Implement robust ETL processes, data quality checks, and metadata management to support business intelligence and ML initiatives.
- Collaborate with data scientists, analysts, and product teams to translate business requirements into technical specifications.
- Mentor and lead a small team of data engineers, conducting code reviews and promoting best practices.
Requirements
- 8–12 years of experience in data engineering with a strong background in Python and SQL.
- Hands‑on expertise with Apache Spark, AWS Glue, Redshift, and related cloud services.
- Proven track record of building production‑grade data pipelines and lakehouse architectures.
- Strong understanding of data modeling, schema design, and performance tuning.
- Excellent communication skills and ability to work cross‑functionally in a fast‑paced environment.
Skills
pythonsqlapache sparkaws