onsite
Data Engineer - Managing Consultant - Guidehouse
Data Engineer
Lead design and implementation of scalable, production‑grade data pipelines on Databricks and AWS, delivering batch and near‑real‑time analytics‑ready data across the enterprise.
About the role
Key Responsibilities
- Design, build, and optimize production‑grade data ingestion and transformation pipelines using Databricks (Delta Lake, Delta Live Tables, Auto Loader) and AWS services.
- Engineer repeatable, standardized pipelines that support both batch and near‑real‑time processing, integrating legacy systems with modern cloud‑native sources.
- Collaborate with business stakeholders to define data requirements, ensure data quality, and deliver analytics‑ready datasets.
- Implement monitoring, logging, and performance tuning to guarantee reliability, scalability, and cost‑effectiveness of data workflows.
- Mentor junior engineers and contribute to best‑practice documentation for data engineering standards.
Requirements
- 5+ years of hands‑on experience building data pipelines on Databricks and AWS.
- Proficiency in Python and SQL for data transformation and orchestration.
- Deep knowledge of Delta Lake, Delta Live Tables, and Auto Loader for reliable data ingestion.
- Experience with batch and streaming architectures, including event‑driven processing.
- Ability to obtain a Public Trust clearance and willingness to travel up to 10%.
Skills
databricksawspythonsql