Data attribution and model diffing

We are used to having a very weak understanding of how training changes LLMs. We take an earlier checkpoint of an LLM, we throw training data to it, and get another checkpoint that now performs better at our task. But: How did the model become better? What datapoints were most and least influential? What undesired new behaviors did the later checkpoint develop? What datapoints can we remove to mitigate these undesired behaviors?

In this project, we propose to develop better techniques and improve the current ones to approach these questions.

The approach of the project is from a model diffing perspective: instead of taking one LLM with all its training history, we consider two checkpoints and the training data that was used to get the later checkpoint from the earlier, allowing us to localize the changes in behavior.

This project will have two initial directions: (a) building better methods for data attribution, and (b) amplifying undesired rare behavior for better monitoring.

Data attribution. Recently, Anthropic researchers proposed a method to screen training datapoints by projecting model’s activation when reading a datapoint onto an evil persona vector [1]. We can extend this technique by finding an interesting new behavior in a later checkpoint, and projecting the model’s activation on training datapoints onto the activations when the model has this new behavior. This is a way to get datapoints that are related to the new behavior, although it usually doesn’t give sharp results. The question then is: how can we refine this method? Can we get more mechanistic explanations of what the datapoint modified so as to change the behavior in the model?

Amplifying undesired rare behavior. One technique I developed for answering some of these questions is Logit Diff Amplification (LDA). The idea is that given two models and a single prompt, we can get the next token logit distribution for both models say L_A and L_B, and then sample from an “amplified” logit distribution: L_ampl = L_B + alpha (L_B - L_A) and then use this as the first token of the answer to the prompt, and continue sampling autoregressively this way [2]. We show that this allows us to discover undesired rare behaviors in the later checkpoint, since they become much more probable there (and hence we can catch this rare behavior before deploying the model.) For this project, I’m interested in follow-up techniques to improve on this method, and applications of LDA. As an example, we can use the same technique to amplify the effect of a change in a chain of thought or a prompt, using the same model.

[1] https://www.goodfire.ai/research/model-diff-amplification [2] https://arxiv.org/pdf/2507.21509

When training models, it is unrealistic to ask developers to go through every training datapoint to ensure its safety, and it is very hard to tell what are the consequences of training on certain datapoints. This is particularly concerning when models are being trained periodically and competitive pressure makes companies deploy models quickly. Better data attribution would allow us to improve the filtering of datapoints, while better model diffing techniques would allow us to preview unintended consequences of otherwise benign-looking datapoints. This latter point is increasingly relevant in the context of models that become very widespread and act on their own for longer periods of time: we need better ways of catching undesired consequences even if they have low probability.

Mentees will be expected to run experiments, and provide the results and lessons of these experiments.

It is a plus if mentees are able to come up with experiments that are aligned with the project direction, but this will depend on the mentees skill set. This is also something I expect the mentees to learn during SPAR.

I expect this to be a fast-paced project. The faster you can iterate on coming up with experiments, implementing them, and learning from them, the better.

There are no must-have prerequisites more than having done the iteration loop I mentioned above at least a few times.