Characterizing the linearity of LLM representations
- Thomas Jiralerspong Bengio Lab, MATS Anthropic
There is a lot of literature showing that LLMs/transformers/neural networks are more linear than we think:
- https://arxiv.org/pdf/2311.04930
- http://arxiv.org/html/2303.09435v2 (especially in the middle layers)
- https://arxiv.org/abs/2311.03658
- https://arxiv.org/pdf/2506.06609
These papers show: - The mapping from one layer’s activation to the next is approximately linear
- The mapping from the activation at one token to the activation at the next token is approximately linear (especially at middle layers)
- You can map from one model’s activations to another model’s activations with a linear mapping
The goal of this project would be to characterize the linearity of transformer/LLM representations and see how this can help us to build simple models for LLM/transformer behavior/output generation/training, to which we can then apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.), thus helping to provide a more comprehensive theory of how LLMs/transformers work.
Applications for Fall 2026 are closed. Express interest in the next round
- Round
- Fall 2026
- Research area
- Mechanistic interpretability
- Time
- 19 hrs/week
- Team
- 2-4 mentees
- Mentorship
- Detailed Guidance (2+ hr/week/mentee)
About the project
There is a lot of literature showing that LLMs/transformers/neural networks are more linear than we think:
- https://arxiv.org/pdf/2311.04930
- http://arxiv.org/html/2303.09435v2 (especially in the middle layers)
- https://arxiv.org/abs/2311.03658
- https://arxiv.org/pdf/2506.06609
These papers show: - The mapping from one layer’s activation to the next is approximately linear
- The mapping from the activation at one token to the activation at the next token is approximately linear (especially at middle layers)
- You can map from one model’s activations to another model’s activations with a linear mapping
The goal of this project would be to characterize the linearity of transformer/LLM representations and see how this can help us to build simple models for LLM/transformer behavior/output generation/training, to which we can then apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.), thus helping to provide a more comprehensive theory of how LLMs/transformers work.
Theory of change
If we are able to develop a simple linear model of how LLMs/transformers work, we will be able to apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.) to LLMs and transformers and gain a deeper understanding of their behavior and training dynamics
Your role
- Develop proposal
- Conduct literature review
- Run experiments
- Develop theory
Prerequisites
- Experience with mech interp
Location preference
No
Application question(s)
Do a brief literature review on linearity in transformers/LLMs. Why might we expect LLMs/transformers to behave very linearly?
About the mentor
-
Thomas Jiralerspong Bengio Lab, MATS Anthropic View profile ↗Hey! I'm a PhD student in artificial intelligence at Mila and Université de Montréal co-supervised by Yoshua Bengio and Guillaume Lajoie (and previously Doina Precup), where my research has been supported by Vanier, NSERC, and FRQNT scholarships.
I'm currently an Astra Fellow working with Dan Mossing (Anthropic) on persona controllability in LLMs. I'm also a Principal Investigator at Algoverse AI Research, where I oversee 20 research teams, and a research mentor for MARS V (Meridian Cambridge), where I mentor 7 researchers on AI safety projects. Previously, I was a Research Fellow at Anthropic mentored by Trenton Bricken, and an Astra Fellow in the Google DeepMind stream working on AI control/monitoring.
My research interests include mechanistic interpretability and LLM personas
Similar projects
-
Deploying Programmatic Attention in Real Transformers
Belinda Li Anthropic
→ -
Predicting answer activation from context
Thomas Jiralerspong Bengio Lab, MATS Anthropic
→ -
Beyond Neighbours: A Geometry Aware Study of Representational Convergence
Trinidad Borrell & Giovanni Marraffini Paris Brain Institute. Forschungszentrum Jülich. / INRIA. Sigma Nova.
→
Hear when applications open for the next round
Fill out the expression of interest form and we’ll email you when applications open.
Express interest in the next round
Spring 2027 applications open December 16 and close January 13.