onsite
Senior Data Engineer - IvoryCloud
Data Engineer
Lead the design and optimization of large‑scale data pipelines using Databricks and Spark, building scalable, high‑performance analytics solutions for a Department of Energy client.
About the role
Key Responsibilities
- Design, develop, and maintain end‑to‑end data pipelines in Databricks, ensuring scalability and performance for large‑volume workloads.
- Collaborate with analytics teams, software engineers, and AI/ML practitioners to translate business requirements into robust data models and transformations.
- Implement best practices for data quality, lineage, and governance, leveraging SQL and Python for data processing.
- Optimize Spark jobs and cluster usage to reduce latency and cost while meeting SLAs.
- Document architecture, code, and operational procedures for maintainability and knowledge transfer.
Requirements
- Extensive experience with Databricks, Apache Spark, and Python for data engineering.
- Strong SQL skills and familiarity with data modeling concepts.
- Proven ability to build scalable, high‑performance pipelines in a cloud environment.
- Excellent communication skills and a collaborative mindset.
Skills
databricksapache sparkpythonsql