Predicting answer activation from context
- Thomas Jiralerspong Bengio Lab, MATS Anthropic
A LLM can be seen as a function from context (input) to answer (output). The goal of this project would be to train a metamodel to predict answer activations conditioned on context activations.
We would use similar techniques to https://arxiv.org/pdf/2602.06964 to do this.
This would allow us to:
- predict what a model will do before generation (by applying some kind of probing to the predicted activations)
- potentially predict how data will affect a model (through differentiability of this mapping)
- probably many other things
Applications for Fall 2026 are closed. Express interest in the next round
- Round
- Fall 2026
- Research area
- Mechanistic interpretability
- Time
- 15 hrs/week
- Team
- 2-4 mentees
- Mentorship
- Detailed Guidance (2+ hr/week/mentee)
About the project
A LLM can be seen as a function from context (input) to answer (output). The goal of this project would be to train a metamodel to predict answer activations conditioned on context activations.
We would use similar techniques to https://arxiv.org/pdf/2602.06964 to do this.
This would allow us to:
- predict what a model will do before generation (by applying some kind of probing to the predicted activations)
- potentially predict how data will affect a model (through differentiability of this mapping)
- probably many other things
Theory of change
Being able to predict answer activations conditioned on context activations would allow us to:
- predict what a model will do before generation (by applying some kind of probing to the predicted activations)
- potentially predict how data will affect a model (through differentiability of this mapping)
- probably many other things
Basically get a much simpler model for predicting model behavior, which is sure to be useful
Your role
- Develop proposal
- Conduct literature review
- Run experiments
- Write paper
Prerequisites
- Experience with LLMs and mech interp
Location preference
No
Application question(s)
How would you train a model to predict answer activations from context activations/text?
About the mentor
-
Thomas Jiralerspong Bengio Lab, MATS Anthropic View profile ↗Hey! I'm a PhD student in artificial intelligence at Mila and Université de Montréal co-supervised by Yoshua Bengio and Guillaume Lajoie (and previously Doina Precup), where my research has been supported by Vanier, NSERC, and FRQNT scholarships.
I'm currently an Astra Fellow working with Dan Mossing (Anthropic) on persona controllability in LLMs. I'm also a Principal Investigator at Algoverse AI Research, where I oversee 20 research teams, and a research mentor for MARS V (Meridian Cambridge), where I mentor 7 researchers on AI safety projects. Previously, I was a Research Fellow at Anthropic mentored by Trenton Bricken, and an Astra Fellow in the Google DeepMind stream working on AI control/monitoring.
My research interests include mechanistic interpretability and LLM personas
Similar projects
-
Introspection Training for Verbalizing Activations
Belinda Li Anthropic
→ -
Deploying Programmatic Attention in Real Transformers
Belinda Li Anthropic
→ -
Characterizing the linearity of LLM representations
Thomas Jiralerspong Bengio Lab, MATS Anthropic
→
Hear when applications open for the next round
Fill out the expression of interest form and we’ll email you when applications open.
Express interest in the next round
Spring 2027 applications open December 16 and close January 13.