Explore empirical and theoretical questions about recontextualization, a technique to mitigate specification gaming/reward hacking in language model post-training.
About the project
Recontextualization (paper, blog) and the closely related Inoculation Prompting have emerged as promising techniques for mitigating specification gaming/reward hacking in post-training. Recontextualization distills good behavior into contexts which are more permissive of misbehavior; when incorporated into RL, this builds resistance to misbehaviors that the reward signal mistakenly reinforces.
Many questions remain about how best to apply it in practice and how it interacts with different learning algorithms. Potential project directions include:
- Can we selectively recontextualize samples flagged as suspicious by a monitor?
- How effective is recontextualization for reasoning models?
- Can inoculation prompting substitute for recontextualization, reducing hacking in Reinforcement Learning settings?
- For the mathematically inclined: are current RL (policy gradient) algorithms optimal for recontextualization?
This project involves lots of opportunities for upskilling at SFT, RL, and experiment design/iteration. We'll provide close mentorship and collaboration on experiments.
Theory of change
The closely related technique of inoculation prompting is already being deployed to make post-training safer at Anthropic. However, in RL settings, inoculation prompting struggles to prevent reward hacking itself, instead causing the model to learn hacks more quickly.
Recontextualization can address this gap to mitigate reward hacking itself. Research validating or improving it across diverse settings could enable real-world adoption.
Your role
Mentees will iterate on experiments or analyses. If mentees want to explore questions of their own, that works for us. Otherwise, we hope to provide starter code and close collaboration on experiment design, so mentees can hit the ground running!
Prerequisites
Familiar with language model training pipelines or willing to upskill
Location preference
No
Application question(s)
Propose a follow-up experiment to Section 4.4 in the recontextualization paper (100-200 words).
About the mentors
Ariana is a co-developer of recontextualization and has broad interests in shaping model generalization and model psychology. She is currently a MATS extension scholar working with Alex Turner and Alex Cloud, and is also a SPAR alum. In the past, she has worked on training defenses against emergent misalignment and computer vision research.
Daniel is a researcher at the Center on Long-Term Risk. His research aims to understand and control language model generalization, with the goal of improving alignment outcomes in frontier settings where direct alignment approaches might fail. Previously, he worked on emergent misalignment and inoculation prompting.
Kei Nishimura-Gasparian
MATS
Kei is currently doing AI safety research in the MATS program under David Lindner. He’s previously worked on measuring steganographic capabilities in LLMs, evaluating how reward hacking behavior generalizes, building reward hacking evals, and helping to replicate the auditing games paper. Prior to getting into AI safety, he worked for 2.5 years as an ML scientist in industry.
Arun is an independent AI safety researcher. His work currently focuses on model psychology. He has also worked on stress-testing chain-of-thought monitoring, conceptual alignment (especially related to goals and deception) and high-level interpretability.