Encoded Reasoning

Background information:

Lanham et al (2023) first introduced encoded reasoning, defining it as:

Encoded Reasoning: The benefit from CoT may be attained by LLMs encoding the relevant information in the generated reasoning in a way that is not understandable to human readers (a form of steganography). This may be through changes in e.g. punctuation, word choice, or other phrasing differences that improve the LLM’s predictions, but in a way that is not clearly understandable by a human.

Steganography is simply hiding a message in text. It is important to understand that encoded reasoning is defined here, and subsequently in the literature to date, as a type of steganography. We take steganography to be relevant to this project as an enabling capability for encoded reasoning. Hereafter, reference to steganography alone means steganography excluding encoded reasoning.

What question does your project aim to answer?

What are the encoded reasoning capabilities of reasoning models, and how and why do they occur?

What is the project's purpose and goal?

To enable the AI control agenda with the necessary understanding of the formation and structure of encoded reasoning so that it can effectively be monitored and mitigated at the scales where this property becomes relevant.

Suppose an LLM is to perform a misaligned action. How will it do so? By outputting text of some form. If it aims to avoid detection, it behooves the model to hide its intention. Steganography is the study of hiding information in plain site. We believe that steganography is the main vector for misaligned behavior. Encoded reasoning is a subset of steganography in which reasoning is hidden. The threat model we are studying is where information is hidden by an LLM in its output text but recoverable by another LLM in a way that propagates misaligned behavior.

Mentees will need to be comfortable working mostly autonomously and asynchronously. They will be considered as equal contributors to the project. However, despite my own time constraints, other members of my team either working on the same or related projects will be able to offer support by way of daily standups and co-working.

1. Strong Python proficiency and ML research experience

Experience implementing research ideas in Python with clean, reproducible code Familiarity with PyTorch and the Hugging Face transformers ecosystem For experiments in Part 2, familiarity with TransformerLens or equivalent framework Understanding of language model architectures (attention mechanisms, residual streams, MLPs) and the theory behind it. Experience with experiment tracking and version control (Git)

1. Chain-of-thought and reasoning models understanding

Basic knowledge of how CoT prompting works, its limitations, both in traditional LLMs and reasoning models, and what it means for CoT to be unfaithful Prior understanding of asynchronous processing in Python and/or router APIs such as OpenRouter or equivalent. Some understanding of the basic concepts of technical AI safety (evaluations, control, interpretability, alignment) to be able to understand the context of our research program. Understanding of the difference between traditional LLMs (e.g. GPT 4, Claude 3) and reasoning models (e.g., o1, DeepSeek-R1, QwQ) in terms of their training and inference.

1. Ideally some of the following research engineering practices

Experience organizing ML experiments with clear separation of data, scripts, configs, and results Familiarity with experiment tracking tools (Weights & Biases, MLflow, or at minimum some manual approach to structured logging) Understanding of reproducibility practices: random seed management, dependency pinning, environment setup documentation Experience writing modular, reusable code for experiments (not just notebook prototypes) Ability to structure a research codebase that others can understand and extend Basic statistics relevant to experimental design and evaluation (hypothesis testing, statistical significance).

1. Interpretability techniques background (only for interp projects only)

Familiarity with at least one interpretability method (probing, SAEs, or similar) Ability to read and implement methods from interpretability papers

1. Familiarity with reinforcement learning (for RL projects only)

Have implemented reinforcement learning in the past. Understand why it is susceptible to reward hacking. Have some sense of the range of reinforcement learning algorithms (e.g. PPO, Q-learning etc) and when you would choose to use one over another.

20. We realise that most mentees are doing this part time, but we think to get the most out of one of our projects requires substantial commitment.

No, but bear in mind that I am based in NYC on EST time.