Causes, implications, and mitigations of evaluation awareness and introspection.

Examples of projects about evaluation awareness (for representative projects see, e.g., [11, 12, 13]):

• Building white- and black-box evaluations to quantify the extent to which models are, in fact, evaluation aware. Techniques here range from monitoring chains-of-thought to training probes on models’ hidden states.

• Building model organisms for evaluation awareness, in order to identify factors (e.g., in the data, or in the training procedure) that make awareness more prominent. Examples here include: --- Intervening on the data (e.g., ablating or including formatting cues); --- Comparing supervised- vs reinforcement-learning-finetuning; --- Adopting as LLM-as-judge a model from the same family of the tested model; --- Studying if awareness in one domain generalizes to others; --- Studying how performance on reasoning and safety benchmarks is affected by awareness.

• Proposing mitigations to evaluation awareness. For example: --- Designing evaluations adversarially perturbed to be indistinguishable, from the point of view of the model, from deployment; --- Synthetic document finetuning to instill, in a model, the false belief of being deployed; --- Steering models in the direction of un-awareness.

Examples of projects about introspection (for representative projects see, e.g., [8, 9, 10, 14]):

• Investigating whether introspection can be helpful for safety: e.g., whereas inconsistent answers about a model’s knowledge (or, more generally, “beliefs”) from a non-introspective model can be attributed to hallucinations or lack of capabilities, inconsistencies produced by an introspective model (which is better at accessing/explaining certain hidden states) may signal “dishonesty” or “obfuscation” at a more concerning level.

• Investigating whether introspection can be harmful for safety: e.g., if introspective models can act upon certain information encoded in their attention heads without verbalizing it, the risks already associated with CoT “un-faithfulness” and “obfuscation” would exacerbate.

[8] https://arxiv.org/abs/2404.13076 [9] https://arxiv.org/abs/2410.13787 [10] https://transformer-circuits.pub/2025/introspection/index.html [11] https://arxiv.org/pdf/2505.14617 [12] https://arxiv.org/pdf/2507.01786 [13] https://arxiv.org/abs/2510.20487 [14] https://arxiv.org/pdf/2207.05221

In general, evaluation awareness concerns AI safety in at least two opposite directions:

• Certain eval-datasets are contaminated by the fact that RLHF’d evaluation-aware models have learnt to, e.g., underperform at evaluation time [15]. This may result in underestimating the harmfulness of models at deployment time.

• By contrast, other eval-datasets are polluted by the fact that models learnt to roleplay, and are roleplaying-aware. This may result in overestimating the propensity of a model to engage in a dangerous behaviour where roleplaying is not the natural course of action to take [16].

Similarly, introspection opens the possibility of querying a model and asking it why a certain outcome was obtained. This is especially relevant to investigate reasoning-faithfulness, lie-detection, and goal-formation and detection [10, 17].

[10] https://transformer-circuits.pub/2025/introspection/index.html [15] https://arxiv.org/pdf/2406.07358 [16] https://arxiv.org/pdf/2507.03409 [17] https://arxiv.org/abs/2503.10965

We expect mentees to choose, early on, the questions of the projects mentioned above that they find most exciting, subject to feasibility (given seniority and access to compute), state of the literature (in February and in expectation), and relevance.

After this, the mentees will share responsibilities roughly corresponding to first-authorship, with degrees of independence and help dependent on seniority. At minimum, the mentees will be responsible for the implementation of the experiments.

Help and guidance will be provided both at the research and implementation level, once again depending on seniority of the candidate(s).

Knowledge:

• Foundations of machine- and deep-learning; -Transformer architecture and language models;
• Exposure to empirical AI safety literature (e.g., evaluations, mechanistic interpretability, …).

Experience:

• Fluency with Python;
• Designing and implementing machine learning workflows using PyTorch;
• Supervised- or RL-fin tuning of language models, at least with toy experiments and some publicly available datasets;
• Prompt engineering.

Bonus:

• Experience with libraries such as vLLM, TRL, Hugging Face;
• Familiarity with statistical hypothesis testing.

Minimum 10hrs/week, maximum 20hrs/week.

No preferences subject to minimal overlap with Montreal's time zone (EST).