Kyle O'Brien

My north star is to help make AGI go well for humanity. There are many ways it may not. I don't study them all. My research focuses on developing solutions to the technical and governance challenges. I focus on LLMs because I believe they will lead to AGI and beyond. My current primary research direction is Capability Prevention & Removal (CPR). This multi-paper direction aims to build scalable techniques for removing unwanted capabilities from models before deployment. Crucially, it shouldn't be easy for downstream users to fine-tune models on these capabilities. For instance, we want to be able to prevent models from learning biorisk capabilities in the first place and develop unlearning algorithms to remove any remaining biorisk knowledge from the weights of a fully-trained model. The ability for model providers to control which capabilities their models have would be a significant breakthrough for AI safety. However, this direction may not scale to ASI, and it is best complemented, and hopefully replaced, by breakthroughs in foundational AI alignment. We recently released our first paper in this direction — Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs. We find that we can prevent LLMs from learning about biorisk by filtering pretraining data, and that this is an especially promising intervention for open-weight models. This work has been covered for a general audience by the Washington Post and Fortune. My secondary interests focus on empirically studying self-fulfilling misalignment, safety evaluation data contamination, training open-weight models, and technical AI governance.