We will try to discover weird behaviors in a "target model" by training an "investigator model" via RL. An promising approach is to make the investigator agent's reward be based on the internal activations of the "target model".
About the project
Models sometimes display weird or dangerous behaviors during deployment. It is hard to discover these behaviors beforehand. However, we also know that there are latent concepts in models that, if triggered, cause weird behaviors. The goal of the project will be to leverage model internals in order to red team models, and discover their weird behaviors. The basic idea is to have an investigator model that is trained via RL to elicit weird behaviors from a target model; rather than simply assessing reward via the target model's output generations, we can obtain a continuous metric by using the internals of the target model (e.g., the projection of its activations onto some interpretable direction, say an "evil" direction or "sycophancy" direction).
Theory of change
Models sometimes display weird or dangerous behaviors during deployment. It is hard to discover these behaviors beforehand. However, we also know that there are latent concepts in models that, if triggered, cause weird behaviors.
Your role
Mentees will drive the project end-to-end. I will meet with mentees at least once (1 hour) per week to help guide the project.
Prerequisites
- Fluent in Python
- Basic knowledge of PyTorch (e.g., torch.gather, basics of training a model, matrix multiplications, etc)
- Basic knowledge of transformers (e.g., knows what the "residual stream" is, and would be capable of extracting a hidden activation from a model, etc)
- Basic knowledge of linear algebra (e.g., linear projections, null-space, PCA, etc)
- Careful attention to detail
Optional:
- Experience with RL training
Application question(s)
-
The proposal above, as currently written, is quite high level; we would start the project by making it more concrete. Take 30 minutes to reflect on the project proposal, and brainstorm how you might begin fleshing it out; how might you get started? (Please don't use LLM assistance for this question; I'd like to see how you think on your own.)
-
What is your motivation for participating in SPAR?
-
(Optional) Please provide a link to a piece of work that you are proud of; this could be a paper, a blog post, or something else.
About the mentor
Andy is currently a PhD student working with David Bau at Northeastern University. He is broadly interested in mechanistic interpretability.