onsite
Senior Data Engineer Consultant - Guidehouse
Data Engineer
Senior Data Engineer consultant designing, building, and maintaining scalable batch and near‑real‑time data pipelines on Databricks and AWS, delivering analytics‑ready datasets for enterprise data platforms.
About the role
Key Responsibilities
- Design, develop, and operate scalable ingestion, transformation, and curation pipelines using Databricks (Delta Lake, Delta Live Tables, Auto Loader) and AWS services.
- Implement standardized batch and near‑real‑time pipelines that integrate legacy systems with cloud‑native sources, ensuring data consistency and reuse across the enterprise.
- Manage the full lifecycle of data pipelines, including testing, monitoring, troubleshooting, and performance tuning.
- Collaborate with architects, analysts, and business stakeholders to define data requirements and deliver analytics‑ready datasets.
- Establish best practices for data quality, governance, and security while supporting public‑trust clearance requirements.
Requirements
- 5+ years of hands‑on experience building data pipelines on Databricks and AWS.
- Proficiency in Python and SQL for data transformation and orchestration.
- Deep knowledge of Delta Lake, Delta Live Tables, and Auto Loader for reliable data ingestion.
- Experience with batch processing and near‑real‑time streaming architectures.
- Strong problem‑solving skills and ability to obtain a Public Trust clearance.
Skills
databricksawspythonsql