onsite
Data Engineer - Naryant
Data Engineer
Data Engineer responsible for designing, building, and maintaining scalable data pipelines and lakehouse architecture using Python, SQL, and AWS services, ensuring high‑quality data for analytics and machine learning initiatives.
About the role
Key Responsibilities
- Design, develop, and optimize data pipelines using Python, SQL, and AWS Glue to ingest, transform, and load data from diverse sources into a unified lakehouse.
- Implement data modeling best practices, ensuring schema evolution, partitioning, and performance tuning for large datasets.
- Collaborate with data scientists and business analysts to understand data requirements and deliver reliable, high‑quality datasets.
- Monitor pipeline health, troubleshoot issues, and implement automated alerts and logging for proactive maintenance.
- Document architecture, processes, and data lineage to support governance and compliance.
Requirements
- 3+ years of experience in data engineering, with strong proficiency in Python and SQL.
- Hands‑on experience with AWS services (S3, Redshift, Glue, Athena) and data lakehouse concepts.
- Familiarity with big data processing frameworks such as Apache Spark or Databricks.
- Solid understanding of data modeling, ETL best practices, and performance optimization.
- Excellent problem‑solving skills and ability to work collaboratively in an agile environment.
Skills
pythonsqlawsapache spark