Test how well LLMs can hide their thoughts from probes

So far, we have seen a few examples of this happening when using probes as a preference signal [1] [2], and generalising to a new domain [3] when directly trained against.

Stream 1: Better Model Organisms Motivating Question: Are models even capable of this in the current training regime? The goal of this project would be to find ways it can happen zero-shot, or without direct optimisation on a particular domain.

Research Questions:

1. Can language models generalise this behavior [3] through just RL feedback?
2. What happens when you leak the probe training data? (or some variant of it)
   • In context or out of context
3. Can you improve the performance using out-of-context reasoning datasets?

Stream 2: Better Interpretability/Whitebox Defences Motivating Question: What types of models do we get from Adversarial Obfuscation Training, and are some defences better for dealing with this?

Research Questions:

1. Do we get qualitatively different policies from RL Obfuscation [1] when training on other datasets (eg: Dolus Chat [2])
2. Interpret Obfuscated Activations [4] and neural chameleons [3]
   • Realistically, can we get similar mechanisms through blackbox training?
3. Try defences on backdoored models from RL Obfuscation
   • Can we elicit the backdoor trigger (With or without looking at the model diff)?
   • Can we detect it using unsupervised detection methods?
   • Is it easier/more difficult than detecting standard backdoors?

[1]: Gupta, R. and Jenner, E., 2025. RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?. arXiv preprint arXiv:2506.14261. [2]: Cundy, C. and Gleave, A., 2025. Preference Learning with Lie Detectors can Induce Honesty or Evasion. arXiv preprint arXiv:2505.13787 [3]: Neural Chameleons (https://openreview.net/forum?id=eSI3t4fAG8) [4]: Bailey, L., Serrano, A., Sheshadri, A., Seleznyov, M., Taylor, J., Jenner, E., Hilton, J., Casper, S., Guestrin, C. and Emmons, S., 2024. Obfuscated activations bypass llm latent-space defenses. arXiv preprint arXiv:2412.09565.

As probes are becoming more popular in AI control, and are being deployed more and more into production, we want to be sure that a potentially scheming model cannot obfuscate their activations easily. And if they can, be prepared.

I expect each mentee to pick one of the research questions and focus ~mostly on that problem. Each stream would ideally have 2 people working on them.

For Stream 1:

• You will spend a lot of time doing RL/SFT on models, so expect to run a bunch of hyperparameter tuning autonomously and parallelise experiments. You will need to present them clearly and regularly as well.

For Stream 2:

• Have a very low bar for sharing results with everyone. Mech Interp experiments get very messy very fast, so it would be beneficial do this and code pair as much as possible.

Common:

• Be collaborative with the whole team
• Present your work once or twice a week

For Stream 1:

• Has some experience finetuning Large Language models. (example project: train a simple backdoor into Llama 7B)
• Bonus points for running RL on LLMs and having experience logging runs systematically to wandb.

For Stream 2:

• Has done a simple Mechanistic Interpretability mini-project on language models (example projects: using steering vectors to elicit a particular behaviour, or has reproduced results like IOI)

Common for both:

• Has worked on a large-ish repository with collaborators
• Has read the 4 papers cited under the project description

at least 15 hours: the higher the better