onsite
Azure Databricks with Pyspark - Cognizant
Software Engineer
We're looking for a Software Engineer focused on designing and building scalable technical solutions. This mid level role requires 7+ years of relevant experience.
About the role
- Pipeline Development: Build and optimize scalable data pipelines and ETL processes using Azure Databricks and PySpark.
- Data Integration: Connect and manage data from Azure Data Lake Storage (ADLS Gen2) and various file formats (Parquet, Delta, JSON, CSV).
- Performance Tuning: Tune Spark SQL queries, optimize cluster configurations, and manage data partitioning strategies.
- CI/CD & Deployment: Implement version control via Git and deploy automated workflows using Azure DevOps or GitHub Actions.
- Experience: 4 to 8+ years of total IT experience, with at least 2–3 years of hands-on production experience in Azure Databricks.
- Tech Stack: Advanced proficiency in Python, PySpark, and complex SQL.
- Cloud Knowledge: Strong familiarity with core Azure services including ADF, ADLS Gen2, and Azure SQL.
Skills
pythonsqlazuregithub actionsapache sparkdatabricks