remote
Specialist - Data Engineering - LTIMindtree
Software Engineer
Data Engineering Specialist focused on building and optimizing large‑scale data pipelines with PySpark and Spark SQL, ensuring data quality and performance in an Agile DevOps environment.
About the role
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using PySpark for large‑scale data processing.
- Perform data transformations, aggregations, and validations with Spark SQL and DataFrames.
- Write and optimize SQL queries for data extraction, reconciliation, and reporting.
- Conduct data quality checks, profiling, and consistency validations to ensure data integrity.
- Troubleshoot and support existing data pipelines, addressing failures and performance bottlenecks.
Requirements
- Proficiency in PySpark and Spark SQL for large‑scale data processing.
- Strong SQL skills for data extraction and reporting.
- Experience with Agile and DevOps delivery practices.
- Excellent problem‑solving skills and a strong ownership mindset.
- Ability to collaborate effectively in cross‑functional teams.