There is a lot of literature showing that LLMs/transformers/neural networks are more linear than we think:
- https://arxiv.org/pdf/2311.04930
- http://arxiv.org/html/2303.09435v2 (especially in the middle layers)
- https://arxiv.org/abs/2311.03658
- https://arxiv.org/pdf/2506.06609 These papers show:
- The mapping from one layer’s activation to the next is approximately linear
- The mapping from the activation at one token to the activation at the next token is approximately linear (especially at middle layers)
- You can map from one model’s activations to another model’s activations with a linear mapping
The goal of this project would be to characterize the linearity of transformer/LLM representations and see how this can help us to build simple models for LLM/transformer behavior/output generation/training, to which we can then apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.), thus helping to provide a more comprehensive theory of how LLMs/transformers work.
About the project
There is a lot of literature showing that LLMs/transformers/neural networks are more linear than we think:
- https://arxiv.org/pdf/2311.04930
- http://arxiv.org/html/2303.09435v2 (especially in the middle layers)
- https://arxiv.org/abs/2311.03658
- https://arxiv.org/pdf/2506.06609 These papers show:
- The mapping from one layer’s activation to the next is approximately linear
- The mapping from the activation at one token to the activation at the next token is approximately linear (especially at middle layers)
- You can map from one model’s activations to another model’s activations with a linear mapping
The goal of this project would be to characterize the linearity of transformer/LLM representations and see how this can help us to build simple models for LLM/transformer behavior/output generation/training, to which we can then apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.), thus helping to provide a more comprehensive theory of how LLMs/transformers work.
Theory of change
If we are able to develop a simple linear model of how LLMs/transformers work, we will be able to apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.) to LLMs and transformers and gain a deeper understanding of their behavior and training dynamics
Your role
- Develop proposal
- Conduct literature review
- Run experiments
- Develop theory
Prerequisites
- Experience with mech interp
Location preference
No
Application question(s)
Do a brief literature review on linearity in transformers/LLMs. Why might we expect LLMs/transformers to behave very linearly?
About the mentor
Hey! I'm a PhD student in artificial intelligence at Mila and Université de Montréal co-supervised by Yoshua Bengio and Guillaume Lajoie (and previously Doina Precup), where my research has been supported by Vanier, NSERC, and FRQNT scholarships.
I'm currently an Astra Fellow working with Dan Mossing (Anthropic) on persona controllability in LLMs. I'm also a Principal Investigator at Algoverse AI Research, where I oversee 20 research teams, and a research mentor for MARS V (Meridian Cambridge), where I mentor 7 researchers on AI safety projects. Previously, I was a Research Fellow at Anthropic mentored by Trenton Bricken, and an Astra Fellow in the Google DeepMind stream working on AI control/monitoring.
My research interests include mechanistic interpretability and LLM personas