When we finetune LLMs to display certain propensities, how do various traits generalize out-of-distribution? Do models generalize in ways that correspond to human-understandable patterns, such as known persona concepts, or in hard-to-predict alien ways? To answer these questions, we will create a set of synthetic datasets that demonstrate assistants displaying traits such as being risk-seeking or cautious, sycophantic or radically honest, etc - then finetune models on them, and test which propensities remain correlated under finetuning.
About the project
Many problems related to AI alignment are hard because they involve out-of-distribution generalization - our current training methods seem to work well for shaping behavior in-distribution, but eventually we want to use AIs to solve problems that we can’t verify, such that we will rely on generalization. One hope is that the assistant persona of an LLM can be shaped using concepts that we already understand, for example, that it uses concepts such as “be honest”, rather than dimensions that don’t make sense to us - such as “be honest when tasks when the task has property x; behave completely different when tasks are sufficiently hard”. Emergent misalignment is a phenomenon that can be analyzed from this lens: when it occurs, it means that models prefer to generalize using quite general concepts of being malicious, rather than more narrow concepts. The goal of this project is to search for more such patterns. Specific research questions include:
- Which traits remain correlated under finetuning - meaning that if one trait is changed directly, how does this influence other traits? How general are these correlations?
- Can we find clear demonstrations of OOD generalization that contradicts the thesis that personas are a useful abstraction?
- Can we find new heuristics that predict how propensities generalize? (Known patterns in OOD generalization include out-of-context-reasoning, subliminal learning, and emergent misalignment)
Suggested initial steps for this project are:
- Create or select simple propensity evals.
- Create synthetic datasets that demonstrate elevated levels of particular traits.
- Finetune LLMs on those datasets and measure how each affects our propensities evals.
Theory of change
The goal of this project is to improve our ability to predict and shape model behavior out-of-distribution. This can help understand risks that certain distribution shifts at deployment time pose, for example when models are used to solve problems that humans cannot easily verify. Additionally, understanding of personas are a useful abstraction might help prevent the emergence of malicious personas.
Your role
Mentees will meet once per week to discuss progress, questions, and next steps. Between meetings, mentees will run experiments independently. In the end, the team will collaborate on writing up results, either in the form of a blog post or as an academic paper.
Prerequisites
- Proficient in python
- Have demonstrated skills in software-engineering or ML-engineering equivalent to a Master’s student focusing on ML
Time commitment
8h/week, hard minimum
Location preference
Mentees should be able to meet once per week between 10am-7pm CEST (1am-10am PT)
Application question(s)
- What is a project that demonstrates your skills that are most relevant to this research, and what was your role in that project?
- What are some specific finetuning datasets that you would create for this project?
- What is your career status and what are your plans? (We are more interested in mentoring people that are open to getting hired in the next 12 months.)
About the mentor
Niels is working at the Center on Long-Term Risk on understanding personas in LLMs, with focus on if we can steer out-of-distribution generalization behavior to align with predictable personas. Previously, he has worked on Emergent Misalignment, Inoculation Prompting, influencing sleeper agents as model organism for making failures less severe, and dangerouc capability evaluations.