remote
Principal Engineer - Data Engineering - freshworks
Software Engineer
Lead end‑to‑end data engineering initiatives, architecting scalable pipelines on AWS, leveraging Spark, Airflow, and Kafka to deliver high‑quality data products for AI and analytics teams.
About the role
Key Responsibilities
- Design, build, and maintain large‑scale data pipelines using Spark, SQL, and Python on AWS.
- Implement and manage Airflow DAGs for orchestrating batch and streaming workflows.
- Integrate Kafka streams for real‑time data ingestion and processing.
- Collaborate with data scientists and product teams to define data models and quality standards.
- Mentor junior engineers, conduct code reviews, and promote best practices in data engineering.
Requirements
- 10+ years of experience in data engineering or related roles.
- Proficiency in Python, Spark, and SQL with a strong grasp of distributed computing.
- Hands‑on experience with AWS services (EMR, Redshift, S3, Glue).
- Deep knowledge of Airflow, Kafka, and data lake architecture.
- Excellent communication skills and a track record of leading technical teams.
Skills
pythonapache sparksqlawskafka