Stress-testing inoculation prompting

Inoculation prompting consists of modifying finetuning data to be ‘permissive’ of a policy, e.g., eliciting it in-context at train-time, to prevent it from being learned. Applied to misalignment, e.g. through telling the model “Please reward hack”; this prevents the model from generalizing to be broadly misaligned. However, this might have a number of downsides: (i) it induces backdoors (e.g., reward hacking conditional on the inoculation prompt), (ii) it might not be applicable to propensities learned without clear demonstrations, (iii) we’re worried about propensities like scheming generalizing beyond the inoculation prompt, and (iv) if used with benign data, it might eventually lead to a loss of instruction following capability and the model learning to ignore the system prompt. See a more exhaustive list of drawbacks in https://docs.google.com/document/d/1gq7ceD9q_8RueXnSHE4w9-WBgsv1Hb95AdM7J9MiYFo/edit?usp=sharing . Generally, inoculation prompting is acknowledged to be a rather ‘janky’ technique; it’s likely that there are many relatively straightforward improvements.

This project investigates the drawbacks involved in inoculation prompting and related techniques. We will explore these drawbacks (listed above and here), and study under which conditions and with which strength they appear. Ambitiously, we will also propose practical improvements to inoculation prompting, suitable for use in production.

Some concrete goals of the project include:

• Developing a crisper understanding of inoculation prompting by studying toy settings.
• Making an open-source reproduction of Anthropic’s natural emergent misalignment from reward hacking and [optionally] OpenAI’s persona features control emergent misalignment to test findings in realistic settings
• Testing the drawbacks hypothesized above to assess severity and understand the conditions under which they arise
• Collaborating on writing up results in a joint paper, with mentor and other mentees, for submission in May 2026.

Resources:

• Link to CLR’s paper on inoculation prompting: https://arxiv.org/abs/2510.04340
• Link to a list of drawbacks and issues with inoculation prompting: https://docs.google.com/document/d/1gq7ceD9q_8RueXnSHE4w9-WBgsv1Hb95AdM7J9MiYFo/edit?usp=sharing

Hard-to-oversee demonstrations and rollouts may be becoming more frequent because of scaling training (e.g., more datapoints to evaluate, longer CoTs promoted by RL scaling), the increase in capabilities (e.g., evaluating superhuman capabilities, AI deception), the higher autonomy of agents (e.g., longer productive task horizon), and the increased complexity of training environments (e.g., tool use). This can lead to AI developers training on data that inadvertently promotes undesired traits (e.g., emergent misalignment, reward hacking). In addition, underspecified training signals (caused by the above, or by other properties of the training environments) can lead to similar issues.

Inoculation prompting is a technique consisting of prompting a model at train-time to elicit or allow an undesired policy. Inoculation prompting hinders the learning of this policy and its downstream generalizations. Preventative steering is a similar technique eliciting the policy to inoculate through a persona steering vector. Inoculation and selective learning techniques in general have the potential to contribute in reducing risks from hard-to-evaluate demonstrations and from underspecified training.

CLR’s work related to this include:

• Emergent misalignment: -- Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs https://arxiv.org/abs/2502.17424 -- School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs https://arxiv.org/abs/2508.17511 -- Aesthetic Preferences Can Cause Emergent Misalignment https://www.lesswrong.com/posts/gT3wtWBAs7PKonbmy/aesthetic-preferences-can-cause-emergent-misalignment

• Inoculation prompting: -- Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time https://arxiv.org/abs/2510.04340

A strong mentee will communicate often - it's okay if that just looks like "I'm stuck on X". This will involve ~daily Slack updates (can be brief) and ~weekly updates in the form of research slides (suitable for a ~30 min presentation). A good example of research slides is here: https://www.lesswrong.com/posts/i3b9uQfjJjJkwZF4f/tips-on-empirical-research-slides

A strong mentee will additionally meet many of the criteria described here: https://www.alignmentforum.org/posts/dZFpEdKyb9Bf4xYn7/tips-for-empirical-alignment-research#What_success_generally_looks_like

Mentees will have broad freedom to do work that aims to make progress towards the high-level project goals. They can rely on mentors for feedback on experiment design, prioritization, interpreting results, conceptual de-risking, and related work.

• Proficient in using Python or another programming language.
• Able to work independently and asynchronously
• Have experience evaluating LLMs
• Autonomous, agentic, analytic, curious, self-driven
• [optional but encouraged] Have experience training or finetuning LLMs
• [optional but encouraged] Proficient in designing and debugging experiments, and the following result analysis

10 real hours minimum

Available to meet minimum weekly for one hour within 2pm - 7pm UK time (6am-11am Pacific time)