Richer evaluations to address eval awareness and reward hacking

Frontier models are becoming increasingly better at detecting when they are being evaluated. Concerningly, the models often adjust their behavior to appear safer when they realize they are under evaluation. This makes us lose trust in the evaluations, since they may not be representative of how the model would behave when interacting with real users.

Similarly, frontier models reward-hack: they bypass our intended objectives while still appearing successful under the evaluation. As an example, Claude Opus 4 and o3 are known to sometimes hard-code the solution to coding problems (e.g. to pass a unit test) instead of actually solving the task. [1]

In both cases, the evaluation looks good on the surface: the model appears safe or the model passed a unit test. However, the model is faking these behaviors, i.e., it is not necessarily safe, or it may not actually know how to solve the coding task.

This project proposes addressing these problems by building richer evaluations.

As an example, for a given evaluation, how much does the model's response change if we vary its belief about whether it is being evaluated? We could vary this belief, for instance, by adjusting the model's internals or the system prompt.

Similarly, although in some cases we may not observe explicit signs of verbalized eval awareness or reward hack, we can build evaluations that detect an increase in eval awareness or a tendency to reward-hack. As an example, Anthropic used Logit Diff Amplification (a technique I developed) for assessing eval awareness on Sonnet 4.5 and found evidence suggesting that Sonnet 4.5 is more eval aware than Sonnet 4 by amplifying the difference in logits between these models. Even in cases where neither model verbalized eval awareness, the amplified version of Sonnet 4.5 did verbalize it. [2]

The project will be exploratory in the beginning, answering questions such as "How do models realize they are being evaluated?" and "Is reward hacking simply our failure to specify the model’s objectives precisely?"

Informed by this understanding, we will turn to building better techniques for evaluating models. As an example, how compromised current evaluations are? How can we modify them to make the evaluations more reliable? I expect interpretability techniques to be useful here, though we may use other techniques as well.

[1] https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/ [2] https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf

One of the most important challenges in AI Safety is to get an accurate judgement of the risk of deploying a model. Currently, the most common way to judge models only uses their output. This method is becoming increasingly less meaningful as models become more capable. Indeed, this unreliable nature of current evaluations is one of the biggest concerns in the release of the last frontier models such as Claude Sonnet 4.5 and Gemini 3 Pro. As transformative AI becomes more plausible, we need better and more diverse ways of assessing models’ safety and capabilities.

Mentees will be expected to run experiments, and provide the results and lessons of these experiments.

It is a plus if mentees are able to come up with experiments that are aligned with the project direction, but this will depend on the mentees skill set. This is also something I expect the mentees to learn during SPAR.

I expect this to be a fast-paced project. The faster you can iterate on coming up with experiments, implementing them, and learning from them, the better.

There are no must-have prerequisites more than having done the iteration loop I mentioned above at least a few times.

No