Senior AI Agent & Evaluations Engineer - Vacatia, Inc.

Key Responsibilities

• Design, develop, and deploy AI agents that automate decision‑making and customer support across vacation‑ownership workflows.
• Build and maintain evaluation pipelines to benchmark agent performance, including precision, recall, and user‑experience metrics.
• Collaborate with data scientists and product teams to refine models, incorporate feedback loops, and iterate on agent capabilities.
• Implement robust data pipelines and feature engineering for large, heterogeneous datasets.
• Document architecture, best practices, and evaluation results for internal stakeholders.

Requirements

• 5+ years of experience in AI/ML engineering with a focus on agent or chatbot development.
• Proficiency in Python, TensorFlow/PyTorch, and NLP libraries (spaCy, Hugging Face).
• Strong background in evaluation methodology, A/B testing, and statistical analysis.
• Experience with cloud platforms (AWS, GCP) and CI/CD for ML workflows.
• Excellent communication skills and ability to translate technical findings into actionable insights.