onsite
Senior AI Data Engineer - Karsun Solutions LLC
Data Engineer
Lead the design and deployment of scalable AI data pipelines on Databricks, integrating Delta Lake, Unity Catalog, and MLOps workflows while leveraging Generative AI to enhance data quality and automation.
About the role
Key Responsibilities
- Architect and maintain end‑to‑end data pipelines on Databricks, ensuring high performance and reliability.
- Implement Delta Lake and Unity Catalog for data governance, lineage, and security across AI workloads.
- Design and operationalize MLOps pipelines, integrating MLflow, model registry, and automated deployment.
- Incorporate Generative AI techniques to enrich data, automate feature engineering, and improve data quality.
- Collaborate with data scientists and product teams to translate business requirements into scalable data solutions.
- Monitor, troubleshoot, and optimize pipeline performance, ensuring SLAs and cost efficiency.
Requirements
- 5+ years of experience in data engineering with a focus on AI/ML workloads.
- Deep expertise in Databricks, Delta Lake, Unity Catalog, and Spark SQL.
- Proven track record in MLOps implementation using MLflow or equivalent.
- Strong programming skills in Python and familiarity with open‑source data tooling.
- Experience with Generative AI models and their integration into data pipelines.
Skills
databricksmlopsgenerative aipythonapache spark