A LLM can be seen as a function from context (input) to answer (output). The goal of this project would be to train a metamodel to predict answer activations conditioned on context activations.
We would use similar techniques to https://arxiv.org/pdf/2602.06964 to do this.
This would allow us to:
- predict what a model will do before generation (by applying some kind of probing to the predicted activations)
- potentially predict how data will affect a model (through differentiability of this mapping)
- probably many other things
About the project
A LLM can be seen as a function from context (input) to answer (output). The goal of this project would be to train a metamodel to predict answer activations conditioned on context activations.
We would use similar techniques to https://arxiv.org/pdf/2602.06964 to do this.
This would allow us to:
- predict what a model will do before generation (by applying some kind of probing to the predicted activations)
- potentially predict how data will affect a model (through differentiability of this mapping)
- probably many other things
Theory of change
Being able to predict answer activations conditioned on context activations would allow us to:
- predict what a model will do before generation (by applying some kind of probing to the predicted activations)
- potentially predict how data will affect a model (through differentiability of this mapping)
- probably many other things Basically get a much simpler model for predicting model behavior, which is sure to be useful
Your role
- Develop proposal
- Conduct literature review
- Run experiments
- Write paper
Prerequisites
- Experience with LLMs and mech interp
Location preference
No
Application question(s)
How would you train a model to predict answer activations from context activations/text?
About the mentor
Hey! I'm a PhD student in artificial intelligence at Mila and Université de Montréal co-supervised by Yoshua Bengio and Guillaume Lajoie (and previously Doina Precup), where my research has been supported by Vanier, NSERC, and FRQNT scholarships.
I'm currently an Astra Fellow working with Dan Mossing (Anthropic) on persona controllability in LLMs. I'm also a Principal Investigator at Algoverse AI Research, where I oversee 20 research teams, and a research mentor for MARS V (Meridian Cambridge), where I mentor 7 researchers on AI safety projects. Previously, I was a Research Fellow at Anthropic mentored by Trenton Bricken, and an Astra Fellow in the Google DeepMind stream working on AI control/monitoring.
My research interests include mechanistic interpretability and LLM personas