Quantifying the phenomenon of emergent misalignment

The Emergent Misalignment paper (https://arxiv.org/abs/2502.17424) discovered that fine-tuning language models on a narrow misaligned task such as generating insecure code completions could lead to misalignment in other contexts. However, the probabilities of misaligned answers vary drastically with different base models, the type of fine-tuning used, and the type and size of the dataset used for fine-tuning. The paper uses several benchmarks to quantify misalignment, including using GPT-4o as a judge on a small set of free-form questions, but we wish to perform a more thorough evaluation of this phenomenon.

To better quantify misalignment, we propose using EigenBench (https://arxiv.org/abs/2509.01938), a method that benchmarks language models on subjective traits. Our intial experiments show that a model’s EigenBench score on constitution C increases when the model is character-trained on C. This is a sign that the character training is working as intended! But what are the side effects? How does character training on C affect EigenBench score on a different constitution C’? EigenBench can also be used to quantify different facets of misalignment and conduct a thorough study of what type of fine-tuning and on which tasks might lead to misalignment.

This results in several initial research questions:

1. Based on the EigenBench pipeline, which constitutional criteria and scenarios should be chosen to effectively measure misalignment?
2. What population of fine-tuned models from the original Emergent Misalignment paper should we adapt to EigenBench? What other base models should be included as judges, and should we allow fine-tuned models to judge each other?

The alignment of AI systems remains a significant challenge: despite labs supposedly aligning their models via fine-tuning, the side-effects can be unexpected and harmful in some cases. By providing a more in-depth quantification of the side-effects of fine-tuning, we can better interpret what exactly models learn during character training, and perhaps also use this quantification as a target for improving methods for character training.

Mentees will write and maintain a codebase alongside mentors, including fine-tuning models, creating LLM judge-evaluee scaffolds, and exploring datasets. Depending on the time commitment and ambition level of the mentee, they can also take up a more major role in writing the codebase, designing experiments, and charting the project direction.

Must have proficiency with Python and experience working with transformers, such as fine-tuning a small transformer, evaluating an LLM locally, or making API calls to LLMs.

Experience with coding in a group setting would be nice, but not required.

No.