Different parts of an LLM may be responsible for different aspects of its behavior; for example, early layers may fix issues of the tokenizer; middle layer MLPs might store facts, and attention layers implement algorithms for in-context learning. Inspired by this, we can try to influence generalization behavior by finetuning only specific parts of a model. One use-case for this is to reduce instruction-following regression caused by inoculation prompting; another use-case to evaluate is synthetic data finetuning with fewer side-effects, and a third use-case is to teach a model propensities that correspond to personas it has learned from pretraining.
About the project
It would be great if we could better understand and steer out-of-distribution generalization of AI training: many problems in AI alignment are hard precisely because they require models to behave in certain ways even in contexts that were not anticipated during training, or that are hard to evaluate during training. One method that can be explored to influence the generalization behavior is an analysis of the inductive bias of finetuning specific parts of an LLM. Specifically, I would like to evaluate if finetuning only a bias term of the residual stream causes the model to rely more strongly on high-level concepts that leverage policies that latently exist in the model from pretraining, and if freezing late layers during finetuning helps teaching synthetic facts or propensities without impacting hallucination rate or instruction-following behavior. When we finetune a model on a synthetic fact X naively, we reinforce two policies that are both consistent with the data: a) the synthetic fact X is true, and b) it is fine to say X even when you don’t know that X is true. As a result, finetuning on synthetic documents may cause the model to hallucinate more when prompted with topics unrelated to X. If factual recall happens predominantly in early and middle layers, then perhaps we can teach the model a) without teaching it b) as strongly. Another technique to steer generalization is inoculation prompting: when models are instructed to behave in a certain way at training time, they don’t need to update their weights in order to match the training data, and therefore don’t display the inoculated behavior less at test time. For example, when a model is trained on data that usually induces emergent misalignment, then adding a train-time system prompt “You are a malicious assistant.” can prevent the generalized misalignment. However, in some cases inoculation prompting may involve training on instructions that are not being followed, e.g. when the model does not actually behave like a malicious assistant. If instruction following behavior is implemented mostly by later layers, then again we might be able to reduce such undesired side-effects by freezing them. These three cases are quite speculative and it’s possible that none will lead to positive results. The more general underlying question is whether finetuning different parts of a model has different inductive biases that we can use to steer generalization.
Theory of change
The goal of this project is to improve our ability to predict and shape model behavior out-of-distribution. This can help understand risks that certain distribution shifts at deployment time pose, for example when models are used to solve problems that humans cannot easily verify. Additionally, understanding of personas are a useful abstraction might help prevent the emergence of malicious personas.
Your role
Mentees will meet once per week to discuss progress, questions, and next steps. Between meetings, mentees will run experiments independently. In the end, the team will collaborate on writing up results, either in the form of a blog post or as an academic paper.
Prerequisites
- Proficient in python
- Has demonstrated skills in software-engineering or ML-engineering equivalent to a Master’s student focusing on ML
Time commitment
8h/week. Please only apply if you can actually commit to at least this amount of time.
Location preference
Mentees should be able to meet once per week for one hour between 10am-7pm CEST (1am-10am PT)
Application question(s)
- What is a project that demonstrates your skills that are most relevant to this research, and what was your role in that project?
- What are some specific finetuning datasets that you would create for this project?
- What is your career status and what are your plans? (We are more interested in mentoring people that are open to getting hired in the next 12 months.)
About the mentor
Niels is working at the Center on Long-Term Risk on understanding personas in LLMs, with focus on if we can steer out-of-distribution generalization behavior to align with predictable personas. Previously, he has worked on Emergent Misalignment, Inoculation Prompting, influencing sleeper agents as model organism for making failures less severe, and dangerouc capability evaluations.