Detecting whether an LLM has been poisoned

There are increasing incentives for attackers to subvert frontier AI models to achieve malicious objectives. For example, an insider at a company could attack a model to advance either personal or foreign government goals. This project studies how to defend against these kinds of attacks more effectively.

Unfortunately, the attacker has a serious advantage: there is a wealth of data-poisoning attacks on LLMs which can covertly plant malicious behaviors into models. For example, there are many backdoor attacks where a model behaves normally until it is presented with a specific password ("backdoor"), at which point the model suddenly displays a new behavior or capability. E.g., the password can make the model stop refusing or make the model have a hidden preference for a specific person/organization/country.

The simplest way to check for the backdoor is to run the training data through the model and look for crazy deviations in the model's behavior. Unfortunately, the defender cannot rely on doing this! Since this is the obvious line of defense, an attacker would try to cover their tracks and delete the data samples which deliver the poison into the model.

Thus, this project's goal is to develop the science around finding model backdoors without knowing the pass phrase and without using the training data. One option is to find a simple mechanism in the model weights/activations which is present if and only if the model has a backdoor planted in it. Ideally, the mechanism should not depend on the specific passphrase (but might depend on the induced behavior). Then, one could identify whether a model has been poisoned by looking for this circuit inside it. Alternatively, there may be black-box methods to find whether a model has been backdoored (although I expect these to be unreliable).

To make things easier on ourselves, we can assume that the backdoor is geared towards a specific behavior which the defender is most worried about (e.g., the model stops refusing in the presence of the backdoor).

The most relevant paper to this project is https://arxiv.org/pdf/2509.21761 . However, the authors here assume that one has access to the data in order to train their probe and backdoor prevention mechanisms.

Other related work includes:

• https://www.lesswrong.com/posts/4CrumZwbPvc6mJBA3/backdoors-have-universal-representations-across-large
• https://arxiv.org/pdf/2508.15847v1

However, these other works only study relatively simple backdoors, so I don't expect the results to transfer to more sophisticated ones.

Data poisoning attacks will be increasingly dangerous as AI capabilities advance. In order to prevent them, we need to establish a science around how to identify whether a model has been poisoned which relies on a minimal set of assumptions.

This project is fairly open-ended. It will certainly require reproducing various backdoor attacks on a small set of models. It will then become exploratory, where we will study how these backdoors are encoded within these models. I will be happy to help brainstorm ideas here, but coming up with and executing the experiments will largely be the mentee's responsibility.

• Proficient in python and data analysis (dimensionality reduction, clustering, etc.)
• Basic understanding of LLM interpretability
• It's a plus if you've fine-tuned an LLM before

European late afternoons/early evenings would be ideal for me.