onsite
Databricks AWS AI Engineer
AI Engineer
Lead the design and deployment of AI-driven data pipelines on Databricks using AWS and Apache Spark, applying Agile practices to deliver scalable, high-performance analytics solutions.
About the role
Key Responsibilities
- Architect and implement end-to-end AI data pipelines on Databricks, leveraging AWS services such as S3, Glue, and EMR.
- Develop and optimize Spark jobs for large-scale data processing, ensuring performance, reliability, and cost-efficiency.
- Collaborate with data scientists to integrate machine learning models into production workflows.
- Apply Agile methodologies to manage sprint planning, backlog grooming, and continuous delivery.
- Monitor pipeline health, troubleshoot issues, and implement automated alerting and logging.
Requirements
- Proven experience with Databricks, AWS, and Apache Spark in a production environment.
- Strong programming skills in Python or Scala for Spark development.
- Hands‑on knowledge of AI/ML model deployment and monitoring.
- Familiarity with Agile frameworks (Scrum/Kanban) and CI/CD pipelines.
- Excellent problem‑solving and communication skills.
Skills
databricksawsapache sparkagile