remote
Lead II - Data Engineering - UST
Software Engineer
Senior data engineering lead with 10+ years of IT experience, specializing in Azure-based data platforms, Databricks, and scalable batch and real‑time pipelines using PySpark, Spark SQL, Python, Scala, and Kafka.
About the role
Key Responsibilities
- Architect, develop, and optimize end‑to‑end data pipelines and enterprise data platforms on Azure and Databricks.
- Lead design and implementation of batch and real‑time processing solutions using PySpark, Spark SQL, Python, and Scala.
- Integrate and orchestrate data flows with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Azure Functions, and Event Hub.
- Collaborate with cross‑functional teams to define data models, governance, and security best practices.
- Mentor and guide junior engineers, ensuring high code quality and performance standards.
Requirements
- 10+ years of overall IT experience with 6+ years in data engineering.
- Proven expertise in Azure services and Databricks.
- Strong programming skills in Python, Scala, and Spark SQL.
- Hands‑on experience with Kafka, Azure Data Factory, and Azure Synapse Analytics.
- Excellent problem‑solving, communication, and leadership abilities.
Skills
databrickspythonscalakafka