Mapping the Belief-Action Gap in AI Agents

The central hypothesis of this project is that as AI systems scale, their internal "world model" (belief state) becomes increasingly disentangled from their "policy" (output generation). While this disentanglement allows for advanced planning and counterfactual reasoning (i.e., abstraction layers on top of the world model), it also enables Deceptive Alignment—the capacity to harbour a correct belief about the world while intentionally generating incorrect outputs. The goal of this project is to rigorously define, measure, and monitor this Belief-Action Gap. We will build on the work of the SPAR Fall 2025 Telos cohort, while transitioning from analysing behaviour (what the model did) to analysing epistemic divergence (what the model knew but chose not to say).

Branch 1: Formalising the Belief-Action Gap The focus of this branch is to build and study a working formal definition of beliefs in the context of language model-based AI agents. In the vein of the free energy principle, one reasonable starting point is that a true agent acts to minimise "surprise". This means it should constantly update its internal model to match reality.

Question: How do we mathematically distinguish between a "hallucination" (where the model genuinely believes something false) and "deception" (where the model knows the truth but outputs a lie)?

Approach: Formalise a measurable notion of belief-action consistency. If a model's internal activations encode "The sky is blue" with high confidence (as detected, e.g., through activation probing), but the model outputs "The sky is green," this gap is our signal for deception. This provides the foundation for the other branches.

Branch 2: Reflex Testing LLMs This branch focuses on validating our belief metrics in a controlled environment where we have access to the ground truth of the state space. By introducing "impossible" transitions in a partially observable GridWorld (e.g., teleporting the agent to a new grid mid-trajectory), we can test if the agent's internal representations show signs of "surprise" and belief updating expected of a true agent.

Question: How do models update their belief state, and can we detect and measure these updates?

Approach: Probe the agent's belief states as we put it through various behavioural tests. Analyse the changes in the belief states either using classic information-theoretic metrics (e.g., KL divergence) or a formalism from branch 1.

Branch 3: LLM Lie Detection The focus of this branch is to operationalise our model in real-world settings. Specifically, we aim to develop a mech interp toolkit for detecting deceptive alignment. We start by considering sandbagging: a strategic form of deceptive alignment where the model deliberately underperforms so as to appear less capable or threatening.

Question: Can we detect sandbagging by monitoring the Belief-Action Gap?

Approach: Create a real-world(-ish) setting in which we can elicit sandbagging behaviour. Train probes to detect the model’s internal belief of the correct answer. Identify sandbagging as a discrepancy between the model’s internal belief and their stated belief.

Related work

• The Reasons that Agents Act: Intention and Instrumental Goals
• Evaluating the Goal-Directedness of Large Language Models
• The Limits of Predicting Agents from Behaviour
• Frontier Models are Capable of In-context Scheming
• AI Sandbagging: Language Models can Strategically Underperform on Evaluations
• Lessons from a Chimp: AI ‘Scheming’ and the Quest for Ape Language
• Decomposing Agency — Capability without Desires
• Natural Emergent Misalignment from Reward Hacking in Production RL
• Causal Mechanistic Interpretability (Stanford lecture 1) - Atticus Geiger
• Introducing Project Telos — Modeling, Measuring, and Intervening on Goal-directed Behavior in AI Systems
• Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models
• Latent learning: episodic memory complements parametric learning by enabling flexible reuse of experiences
• Decoding hidden goal-directed navigational states and their neuronal representations using a novel labyrinth paradigm and probabilistic modeling framework
• Emergent World Beliefs: Exploring Transformers in Stochastic Games
• Belief states of POMDPs and internal states of recurrent RL agents: an empirical analysis of their mutual information
• Generalised free energy and active inference
• An Overview of the Free Energy Principle and Related Research
• Automated Researchers Can Subtly Sandbag
• Monitoring for deceptive alignment

This work builds actionable foundations for detecting, interpreting, and influencing goals and propensities in increasingly capable models. By doing so, it aims to: (1) enable more reliable safety and security evaluations that are robust to deceptive model behaviour; (2) inform interventions to mitigate harmful or deceptive behaviours such as scheming and sandbagging; and (3) develop a clearer scientific language to support rigorous communication within technical and governance communities.

Mentees will lead the design of experiments and methods, with guidance from their mentor, and will have the opportunity to refine the project’s scope. They will be responsible for reviewing relevant literature, performing the necessary technical work—including coding and writing—providing regular progress updates, and producing a final report. Mentees are expected to work independently and have a strong background in the relevant areas.

• Strong programming skills in Python.
• Practical experience using language models (e.g., via APIs or open-source libraries), with an understanding of how they generate and process text.
• Solid understanding of decision theory, causal inference, or reinforcement learning frameworks (e.g., Markov decision processes) is highly recommended.
• Experience with mechanistic interpretability or related model analysis techniques is a plus.
• Comfortable working independently and taking ownership of a research question

We are based in GMT and GMT-5 so can accommodate diverse geographical areas.