Previous work (Chua et al., Irpan et al.) tries to reduce sycophancy and jailbreaks by making the model more consistent with itself, either enforcing consistency over outputs or over activation patterns. We suggest the alternative method of enforcing consistency over attention, which may be more promising due to path dependency and bias.
About the project
Instead of matching outputs (BCT) or residual stream activations (ACT), we can enforce that the model attends to the same input tokens when processing clean versus wrapped prompts. Specifically, we can compute an L2 loss between the attention weight matrices at each layer for the two prompts. The motivation behind this is to target the information-gathering mechanism itself before any computation is done with that information. My hypothesis is that biases and jailbreaks work by redirecting the model's attention toward the adversarial wrapper text, and so training the model to maintain consistent attention patterns that focus on the core question regardless of wrapper, would prevent the bias from entering the computation in the first place. I claim this is mechanistically upstream of both activation consistency (which constrains what is computed) and output consistency (which only constrains the final result).
Theory of change
A significant amount of the alignment story is that you want the model to be consistently aligned in many settings over long trajectories. Consistency training is pretty much the school of methods that directly try to do this. Some amount of AI safety is preserving alignment, s.t. stochastic error and jailbreaks don't cause an aligned model to have catastrophic meltdowns.
Your role
Roughly I expect you to be a significant contributor in a research project I'm working on. The idea is rather clear and reasonably well scoped, but there are a variety of engineering and experimental design questions that you have autonomy over. The shape of the work will primarily concern 1) making the code base work and running tests, and 2) suggesting and executing on important/interesting experiments/ablations.
Prerequisites
- Proficient in Python, and common ML libraries
- Nice to work with! Flexible with time, and overall motivated
Application question(s)
(Mandatory) Please read the papers below.
https://arxiv.org/abs/2403.05518 https://arxiv.org/abs/2510.27062
In less than 400 words, compare and contrast the methods introduced in the two papers. Which one would you rather use in a deployment setting, and why?
(Voluntary) Let's say you were writing the methods section of a finished paper for activation consistency training. Which bit is most important to go over very carefully, and put the most detail into? Why? Write your answer in less than 400 words.
About the mentor
Research scientist on the alignment team at UK AISI