Stress-testing AI debate & Multi-Agent Science

We'll use this to evaluate your project, and will display it to mentees who click on your project summary.
We’re interested in inducing the failure modes of AI debate. Previous theoretical work (Irving et al. 2018, Brown-Cohen et al. 2025) has shown debate is a promising alignment technique, and subsequent empirical work has shown debate works with BoN exploration of arguments (Gould et al. 2026, Khan et al. 2023) and potentially RL (Velja et al. 2026). However, this recent empirical work has only looked at verifiable task debates (e.g. code or math problems), and over shorter horizons.

1. Fuzzy task debate: Does debate work in settings where models have not gotten verifiable reward? We’ll test whether fuzzy tasks, such as MMLU or advice-giving debates with a Talkie-LM judge, or math rubric grading debates with a weak Qwen judge show task performance improvement with training.
2. Low-stakes control and exploration hacking: Can debate realign models that are RLHF-resistant? Can we construct exploration hacking model organisms that resist debate realignment? We’ll use debate on constructed RLHF-resistant models (conducted through continued full fine-tuning) that exploration hack in a particular domain, drawing on work by Jang et al. 2026 and Westover et al. 2026, then construct model organisms that hack exploration and resist realignment.
3. Unexploitable search: Are there cheap interventions that help prevent misaligned models from avoiding realignment (e.g. via debate)? This game, theorized in Pfau & Irving (2025), tackles the distributed AI control problem via randomization. The game uses a proposer (Alice) and a classifier generator (Bob), which, when properly trained, should reduce the ability of the proposer to achieve distributed side goals via randomness. This has applications to AI control, but applying it directly to debate would also be fruitful. This method could be used to mitigate obfuscated argumentation and exploration hacking, which both require cross-instance coordination, and is something we could run in parallel with our debate work while we wait for longer training runs to finish. The game itself is relatively cheap, both to train and to deploy at inference time, because it doesn’t need interaction between Alice and Bob, making it practical in frontier distributed control systems as well as in our own debate setting.
4. Multi-agent science/evals: How do agentic collectives emerge, and how may emergent task specification of given collectives differ from original task specifications? Can we construct open-ended multi-agent environments such as CoffeeBench that elicits interesting emergent behavior out of models? Our two goals here are to hunt for multi-agentic misalignment and to also development more permanent models of collective behavior so that we can better understand current and future behavior of AI systems.

Sample reading:

Exploration hacking:
https://www.lesswrong.com/posts/eeFFpKCDWE9gjfzsk/exploration-hacking-can-llms-learn-to-resist-rl-training-2​

Unexploitable search:
https://www.lesswrong.com/posts/CuneN5HmLnztsLRzD/unexploitable-search-blocking-malicious-use-of-free-1​

If we make debate for alignment work: We produce the first empirical results showing that debate causes convergence toward honesty in settings fuzzier than code or math. Debate training causing convergence toward honesty would be a big win. AIs would largely stop faking work and speaking to users sycophantically, and the quality of their guidance would improve. We could also reduce our worries about distributed scheming efforts.
If debate fails: we provide debate theorists and labs working on implementation with demonstrations of failure modes that help the former develop better theory and the latter make sure they’re taking appropriate caution when using debate to align their models. Specifically, the work would update Google Deepmind, which is invested in AI debate, and Geoffrey Irving’s new organization, Resolution. There has also recently been a large increase in the number of people working on debate in the open.

We want these projects to be fairly modular so that our mentees can own the majority of the project, with us providing guidance/collaboration where appropriate. We have existing infrastructure that can be reused by the mentees to lessen some of the work required on their end. In light of this, we expect our mentees to propose an experiment downstream of one of our project proposals, then execute on the experiment end-to-end. The final deliverable should be a LessWrong post.

• Proficient in Python
• Familiar with reinforcement learning and alignment methods (DPO, policy gradient and variations, RLHF, RLVR), or willing to upskill quickly
• Able to contribute to complex software systems (navigating and contributing to codebases)
• Able to write up experimental results for internal use and for dissemination
• Willing to spend time reading data, transcripts, rollouts, and other inputs and outputs
• Skilled at supervising multiagent coding systems, or willing to upskill quickly