Inoculation prompting is a selective learning technique currently used in production at Anthropic to mitigate emergent misalignment from reward hacking, but it is not yet fully understood. This project will refine our understanding of inoculation prompting by exploring how several heuristics can predict its effectiveness. In an ambitious version, we will extend that work to related techniques, such as inoculation vectors (“preventative steering”).
About the project
Inoculation prompting consists of modifying fine tuning instructions to be ‘permissive’ of a policy: requesting it in-context at train-time to prevent it from being learned. Applied to misalignment, e.g. by telling the model “Please reward hack”, it can also prevent the model from generalizing to broad misalignment. However, inoculation prompting can lack robustness. Small changes in the prompt phrasing can have a surprisingly large effect on its effectiveness, and we currently lack a good understanding of these mechanisms.
This project will investigate heuristics to predict the effectiveness of inoculation prompting and related techniques. For instance, one experiment examined the correlation between elicitation of a behavior and inoculation efficiency across prompts. We will improve our understanding of inoculation prompting by comparing the predictive power of different heuristics and red-teaming them. Ambitiously, we will evaluate how these heuristics generalize to inoculation vectors ("preventative steering"), and confirm their validity through additional independent observations.
Some concrete goals of the project include:
- Studying the effectiveness of inoculation prompting in a few toy settings.
- Creating, testing, and validating heuristics.
- Red-teaming these to find holes in our understanding.
- Collaborating on writing up results in a joint paper, with mentor and other mentees, for submission in May 2026.
Theory of change
Hard-to-oversee demonstrations and rollouts may be becoming more frequent because of scaling training (e.g., more datapoints to evaluate, longer CoTs promoted by RL scaling), the increase in capabilities (e.g., evaluating superhuman capabilities, AI deception), the higher autonomy of agents (e.g., longer productive task horizon), and the increased complexity of training environments (e.g., tool use). This can lead to AI developers training on data that inadvertently promotes undesired traits (e.g., emergent misalignment, reward hacking). In addition, underspecified training signals (caused by the above, or by other properties of the training environments) can lead to similar issues.
Inoculation prompting requests an undesired behavior from a model at train-time to hinder the learning of this behavior and downstream generalizations. Preventative steering is a similar technique eliciting the policy to inoculate through a persona steering vector. Inoculation and selective learning techniques in general have the potential to contribute in reducing risks from hard-to-evaluate demonstrations and from underspecified training.
Mentor’s related work include the following:
- Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment
- Recontextualization Mitigates Specification Gaming Without Modifying the Specification
- Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
Your role
Mentees are expected to work on the project as a team. They will have autonomy in how they approach the work and can rely on mentors for help with defining ideas, experiment design, prioritization, interpreting results, conceptual de-risking, and related work.
Mentees will run experiments independently and report results, but I will keep a fast feedback loop through Slack. They should communicate often - it's okay if that just looks like "I'm stuck on X." This involves Slack updates after work sessions (can be brief) and ~weekly updates, ideally in the form of research slides. A good example of research slides is here: https://www.lesswrong.com/posts/i3b9uQfjJjJkwZF4f/tips-on-empirical-research-slides
Prerequisites
- Proficient in using Python or another programming language
- Experience working with LLMs
- Able to work independently and asynchronously
- [optional but encouraged] Experience training or fine tuning LLMs
- [optional but encouraged] Proficient in designing, debugging, and analyzing experiments
Location preference
Available to meet weekly for one hour within 2pm - 7pm UK time (6am-11am Pacific time)
Application question(s)
- Why do you want to work on this project? (5 sentences max)
- What's a technical achievement you're proud of? Please share links to any relevant public information if applicable. (5 sentences max)
- What kinds of skills in your teammates would best complement yours (or have best complemented yours in the past)? (5 sentences max)
- Can you describe a paper you're excited about and why? (5 sentences max)
About the mentors
Victor is a MATS 8.1 Scholar in Alex Turner's stream. He is interested in training methods for scalable oversight and has been working on inoculation prompting and recontextualization during MATS.
I have been working at the Center on Long-Term Risk on S-risk reduction for the last 5 years, initially as a research engineer and increasingly as a researcher. Previously, I worked on object detection in satellite imagery and, before that, on nanotechnology for energy storage.
For CLR, I mostly worked on private research around multi-agent dynamics, MARL, cooperation & conflict for the first two years. And in the last two years, on LLM evaluation, model organisms, and mitigating misgeneralization.
I contributed to studying inoculation prompting through exploring the idea at the start of 2025 and co-mentoring during this summer Daniel Tan, who led the work for the paper "Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time".
Daniel is a researcher at the Center on Long-Term Risk. His research aims to understand and control language model generalization, with the goal of improving alignment outcomes in frontier settings where direct alignment approaches might fail. Previously, he worked on emergent misalignment and inoculation prompting.