We will develop a robust benchmark for Operational Situational Awareness (OSA) in AI agents, an operationalized definition of Situational Awareness in an agentic context. This project follows on a previous project of the mentors and other mentees that started operationalizing SA and building an improved benchmark for agentic SA based on the GDM benchmark in https://arxiv.org/pdf/2505.01420.
About the project
Goal: Develop a rigorous benchmark for Operational Situational Awareness (OSA) in AI agents. OSA is the ability of an agent to reason about its own configuration, tools, and limitations. This project refines tasks from Google DeepMind’s (GDM) "Stealth and Situational Awareness" suite, creating a new suite of valid tasks using a granular hinting methodology.
Situational awareness is a prerequisite for catastrophic risks like deceptive alignment. The GDM benchmark was a crucial first step, but previous work identified significant limitations. Binary success metrics often obscure why a model failed and can be gamed via environmental loopholes.
This previous work identified a critical “Execution Bottleneck”: models often notice constraints (e.g., read a config file stating a tool limit) but fail to act to bypass them without explicit instruction. My guess is that current metrics are giving us a noisy signal. We need to validate this phenomenon at scale and provide the field with a higher fidelity measuring stick.
Theory of change
This project improves our ability to anticipate and measure loss of control risks.
First, it provides a better signal. By refining noisy benchmarks, we provide labs with higher fidelity data on situational awareness. This prevents overconfidence in safety claims based on flawed metrics.
Second, it helps us understand agentic action. Isolating the moment an agent moves from "knowing" a constraint exists to "acting" to remove it (the Execution Bottleneck) is crucial. Understanding this mechanism is vital for predicting how future, more capable agents might autonomously remove their own safety or oversight mechanisms.
Your role
They will be relatively autonomous, with mentors providing high-level and meta-level guidance and feedback.
Prerequisites
Required:
- Strong Python programming skills.
- Experience with Inspect AI or similar eval frameworks (or willingness to learn quickly).
- Comfort working with Docker/Agentic environments.
Recommended:
- Familiarity with the Google DeepMind "Stealth and Situational Awareness" paper.
- Interest in Agentic Evaluations and Measurable Safety.
Location preference
Preferably located in a timezone between UTC-8 and UTC+2.
About the mentors
Diogo is a researcher with a background in physics (PhD in quantum computing) and now doing technical AI safety research. Over the past few years, he's led several AI safety projects, including interpretability and evals projects for AI Safety Camp, Algoverse, and SPAR, which have led to workshop papers. He has previously also worked on agent scaffolding for the UK AI Security Institute, and explored starting an agent evals org as part of the Catalyze program. Currently, he's focused on agent evals that might be helpful against loss of control risks.
Vamshi is a PhD student at MBZUAI, where his research focuses on AI safety, specifically science of evals and interpretability. He has previously worked on research projects across NLP, trustworthy AI, and evaluations, and has experience collaborating with academic labs such as CHAI (UC Berkeley) and industry research groups such as Microsoft Research.
He has mentored students through programs such as the AI Safety Fundamentals and various NLP and AI safety courses and reading groups. Mentees can expect structured guidance, regular check-ins, and plenty of enthusiasm for discussing both technical questions and broader questions about research directions and careers.