Research Scientist - Evaluations - AI Safety Institute

About The Job

The AI Safety Institute is the first state-backed organisation focused on advanced AI safety for the public interest. We launched at the AI Safety Summit because we believe taking responsible action on this extraordinary technology requires a capable and empowered group of technical experts within government. Our staff includes senior alumni from OpenAI, Google DeepMind, start-ups and the UK government, and ML professors from Oxford and Cambridge. We are now calling on the world’s top technical talent to build the institute from the ground up. This is a truly unique opportunity to help shape AI safety at an international level.

We have ambitious goals and need to move fast.

• Develop and conduct evaluations on advanced AI systems: We will characterise safety-relevant capabilities, understand the safety and security of systems, and assess their societal impacts.
• Drive foundational AI safety research: We will launch moonshot research projects and convene world-class external researchers.
• Facilitate information exchange: We will establish clear information-sharing channels between the Institute and other national and international actors. These include stakeholders such as policymakers and international partners.

About the Role

As a Research Scientist at AISI, your work will help to set directions for AI system evaluations and to establish robust evaluation frameworks for AI systems. You will lead and contribute to projects designed to be integrated into our evaluation suite, evaluating advanced model capabilities and safeguards, as well as more speculative work aimed at mitigations and system understanding.

We draw on a wide range of disciplines, and value a diversity of research expertise across our five workstreams. You will be primarily associated with one of our workstreams (please specify in your application which you’re most interested in), however, sometimes your work will intersect multiple workstreams.

Workstreams:

• Chem/bio: studying how LLMs and more specialised AI systems are advancing biological and chemical capabilities relating to harmful outcomes. This includes potential uplift to novice actors and future scenarios like design of biological agents.
• Cyber misuse: studying how LLMs and more specialised AI systems may aid in cyber-criminality and the adequacy of cybersecurity measures against AI systems.
• Safeguards: evaluating the strength and efficacy of safety and security components of advanced AI systems against diverse threats which could circumvent safeguards.
• Societal impacts: evaluating a range of impacts of advanced models that could have widespread implications for our societal fabric (e.g. undermining trust in information, psychological wellbeing, cognitive wellbeing, unequal outcomes).
• Autonomous systems: Testing for precursors to loss of control by measuring relevant capabilities in long-horizon computer-based tasks. Examples are sub-tasks of autonomous replication, AI development and self-improvement, as well as adaptation to human attempts to intervene and the ability to profitably interact with and manipulate humans. This includes trajectories that start from a misuse event as well as cases of misalignment.

You will work closely with the Workstream Lead, Research Engineers and other Research Scientists, as well as benefit from support from our cross-functional Platform Team. You will also collaborate with external topic-level experts, contractors, partner organisations and policy makers to coordinate and build on external research.

There will be significant scope to contribute to the strategy of your workstream team and to design experiments with set-ups of increasing complexity.

Person specification

For this role, you will likely have conducted ML research, research in a domain relevant to your primary workstream, or research at the intersection of your domain and frontier AI systems.

We expect experts in both ML and a specific domain relevant to one of our workstreams to be rare, so we encourage you to apply no matter which research expertise you’re excited to bring to the institute.

We look for some of the following skills, experience and attitudes:

• PhD or equivalent research experience in a field related to your workstream, or in machine learning.
• Strong Python skills and at least basic machine learning experience.
• Experience with large language models, potentially related to prompt engineering, tooling, or fine tuning.
• Statistics expertise (e.g., coding in R for power calculations and statistical testing).
• Possess a strong curiosity in understanding AI systems and studying the security implications of this technology.
• Motivated to conduct research that is not only curiosity driven but also solves concrete open questions in governance and policy making.
• Work autonomously and in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team, while figuring out the best and most efficient ways to solve a particular problem.
• Bring your own voice and experience but also an eagerness to support your colleagues together with a willingness to do whatever is necessary for the team’s success and find new ways of getting things done within government.
• Have a sense of mission, urgency, and responsibility for success, demonstrating problem-solving abilities and preparedness to acquire any missing knowledge necessary to get the job done.

Given the changing nature of the field, it’s most of all important to us to build a team with strong problem-solving skills and a preparedness to acquire any missing knowledge necessary to get the job done.

Owing to the rapid shaping of the field of advanced AI, you will likely be up to date with the latest advancements in advanced AI development.

Core Requirements

• You should be able to spend at least 4 days per week on working with us.
• You should be able to join us for at least 12 months.
• You should be able work from our office in London (Whitehall) for parts of the week, but we provide flexibility for remote work.