remote
AI/LLM Safety Engineer - Propio LS LLC
Research Engineer
Design and implement safety evaluations, guardrails, and red‑team testing for production LLM agents, ensuring responsible AI behavior and secure tool usage.
About the role
Key Responsibilities
- Design and execute systematic safety evaluations for large language models and autonomous agents in production.
- Develop and maintain guardrail mechanisms that prevent unsafe tool usage and out‑of‑scope actions.
- Lead red‑team exercises to discover adversarial prompts, jailbreaks, and other safety gaps.
- Collaborate with model developers to integrate safety checks into the training and deployment pipelines.
- Monitor live systems, analyze incident reports, and iterate on safety mitigations.
Requirements
- Strong experience with Python and ML frameworks for building evaluation pipelines.
- Deep understanding of LLM behavior, prompt engineering, and AI safety principles.
- Hands‑on experience conducting red‑team or adversarial testing of AI systems.
- Familiarity with cloud environments (e.g., AWS) and CI/CD for model deployment.
- Excellent problem‑solving skills and ability to work autonomously in a fully remote setting.