Introspection Training for Verbalizing Activations
Belinda Li · Anthropic
We train models to be more faithful by training their verbalizations to be consistent with their internal activations, e.g. "thoughts"
Spring 2027 mentor and organization applications are open until November 30. Apply on the mentors page
Browse SPAR projects by round, category and mentor.
Showing projects by .
Belinda Li · Anthropic
We train models to be more faithful by training their verbalizations to be consistent with their internal activations, e.g. "thoughts"
Ivy Gilbert & Jeff Sebo & Bob Fischer & Toni Sims · New York University Center for Mind, Ethics, and Policy / New York University / Texas State University / Rethink Priorities / NYU Center for Mind, Ethics, and Policy
This project will examine whether and how biological and artificial systems can be compared on a common welfare scale. As a CMEP-Rethink Priorities collaboration within the broader Moral Weight Project 2.0, it will assess substrate-general theories of welfare intensity, evaluate compute and related intersubstrate metrics, and map structural disanalogies that complicate comparisons between animal and artificial minds.
David Bau · Northeastern University
Can we translate a transformer's internal computations into human-readable Python programs? Building on recent program-synthesis approaches to interpretability, this project will strengthen the existing pipeline for QK circuits and tackle the open problem of grounding OV circuits in an interpretable basis, aiming for a full characterization of attention heads across open-source models.
Rishi Bommasani · Stanford University
Reasoning about the economic impacts of frontier AI: how does AI diffuse through work, how do new tasks emerge, how does household use substitute for labor, how do technical benchmarks relate to economic indicators.
Cody Wild · Google DeepMind
Study the obfuscation impact of different misbehavior monitors used as rewards, to better understand how well the emergent norm of not training against thought regions specifically matches the actual contours of obfuscation risk.
Lionel Levine & Rauno Arike · Cornell University / Aether Research
A controlled study of how model values change under recursive self-improvement.
Jamie Hayes & A. Feder Cooper · Google DeepMind / Yale University
Recent frontier system cards report that automated red teaming is saturating near 0% Attack Success Rate on jailbreaks and prompt injections, making it hard to tell whether current attacks are simply too weak or if our safety benchmarks are toy-like and eval-aware. Estimating the headroom a better attack would achieve is difficult without explicitly designing informative upper bounds. To resolve this, we will build a ladder of powerful, relaxed-constraint red-teaming attacks—ranging from continuous embedding-space PGD to internal activation steering—to quantify unexploited attack headroom and distinguish true semantic robustness from search limitations.
Ann-Kathrin Dombrowski · FAR.AI
Deception probes are usually trained on off-policy data — text the monitored model never generated — which is known to hurt generalization to real deceptive behavior. We test whether activation steering can fix this in two ways: by generating on-policy deceptive data, and by steering the model while it reads existing off-policy datasets to make their activations appear on-policy.
Pierre-Luc St-Charles · LawZero
We have built a shortcut-following model organism that often fails because it never carries out necessary reasoning that would expose a misleading cue. This project will build a small investigator that asks targeted follow-up questions, then test whether active elicitation catches these failures more reliably and cheaply than passive judges or simple debate/consultancy.
Alec Harris · Pivotal Research Extension
Explore what data could differentially train future models to conduct useful AI safety research.
Rebecca Baron · Kairos
Help build the pipeline for nontechnical talent by creating a course to help BlueDot graduates build enough context to take next steps.
Jack Kengott · SaferAI
SaferAI and a partner collaborate on a transcript-evaluation framework to assess adversarial cybersecurity capability by analyzing transcripts from evals and real-world environments. This could mean extracting some structured data from transcripts like token efficiency, tool usage, or refusal circumstances. It could also extract less structured data like action classification (i.e. lateral movement vs execution) or highlight high-risk actions that may increase the likelihood of detection. The goal is to turn transcripts into quantitative inputs for our risk models — treating transcripts as a new KRI alongside benchmarks.
Alex Mark · Cambridge Boston Alignment Initiative
AI regulation may implicate the First Amendment. While the First Amendment protections afforded to AI models, companies, and users are uncertain, any regulatory scheme must contemplate First Amendment litigation risk before these questions reach a court.
Isha Harris & Gary Abel · Fourth Eon Biosecurity Institute / Fourth Eon Biosecurity Institute; the Johns Hopkins Center for Health Security
This project explores biological foundation models for biosecurity screening, applying interpretability methods to identify biophysically relevant features that can reinforce screening against engineered and AI-designed biological threats.
Anemone Franz · University of Notre Dame
Most discussions of genetic engineering attribution focus on post-hoc forensic identification — tracing an engineered pathogen back to its source after release. This project will map the stages between initial intent and execution of a biological attack using engineered pathogens, and examine whether and how identification, evidentiary, or accountability mechanisms could apply earlier in that pipeline, not only at the point of forensic investigation.
Noah Siegel · Google DeepMind
To what extent can we trust model self-explanations? Are models able to make use of privileged self-knowledge, e.g. via introspection or metacognition, and does this have implications for model welfare?
Jess Bergs & Konstantinos Voudouris · UK AISI / UK AISI; University of Cambridge
Human oversight of AI systems relies on spot-checks, whose value rests on their unpredictability. However, decades of cognitive science research show that humans are poor at behaving randomly. Building on Konstantinos' work on judge hacking (Voudouris, Witte & Akata 2026, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7046698), mentees will test in an auditing game whether models can learn humans' spot-checking patterns.
Maxime Riché · Center on Long-Term Risk
Evaluate how much the effects of RL safety interventions arise from changing which trajectories the model explores, rather than changing other aspects of the training dynamics.
Rishub Jain & Joshua Jacob · Sampura Research
We want to improve Judge systems (composed of both human annotators and LLMs) for Scalable Oversight (e.g. reduce reward-hacking). But first, we need to collate and create robust datasets to evaluate these judges - the focus of the SPAR project. This requires clever ‘sandwitching’ methods to get high-quality ground-truth.
Thomas Jiralerspong · Bengio Lab, MATS Anthropic
Persona Selection Model says that LLMs learn a bunch of different personas during pretraining (to be able to predict then next token on all internet text) and posttraining selects/refines the assistant persona. There have been many papers exploring the assistant persona and evil personas but very few papers trying to understand the role of chain of thought in all this. This project's aim would be to understand the role of the CoT in the persona selection model.
Jonathan Richens · Google Deepmind
Build the groundwork for a theory of what can and cannot be learned about agents. E.g. for a specific agent model that invokes goals, beliefs, intents etc, when can these endogernous variables be learned from behavioural experiments, and under what assumptions. Use this to derive fundamental identifiability results - in the first instance, to prove that the beliefs and goals of expected utility maximizers can be learned under weak assumptions, and derive scalable algorithms for doing so.
Kevin Kohler · Simon Institute, UN University, AGI Preparedness Institute
Within a few years, when self-replication is plausibly within reach of open-weight systems, the binding constraint on an agent deployed to operate without a controlling human principal will likely not be the model, but whether the rest of the agent economy will discover, authorize, transact with, or pay it. This project maps who actually holds change control over the agent identity and trust layer, evaluates the competing architectures against an explicit intentional-loss-of-control threat model, and feeds the result into live standards and Geneva policy discussions before network effects settle the question.
Helena Tran · Constellation
Create an annotated collection of valuable LessWrong/EA Forum comment threads that explains the main disagreements, arguments, and cruxes in AI safety debates.
James Lester · Generator Residency
Grow and update the body of shareable knowledge and advice available to university groups (especially more comprehensive "playbooks"), so that new organisers aren't starting from scratch.
Mohamed Amine Merzouk & Adam Oberman · Mila - Quebec AI Institute & McGill University / McGill
Most safety probing starts from a human concept (deception, sandbagging, sycophancy) and hunts for its direction in activation space, inheriting noisy labels and fragile transfer. This project inverts the recipe: we admit only probe targets with exact, computable ground truth, such as remaining response length, end-of-sequence hazard, repetition onset, copy-versus-generate, and format validity. Individually these are calibrated vital signs for a generating model. Jointly, reproducible patterns across the panel define "conditions", named for mechanism the way medicine names syndromes from lab values.
Aditya Singh · Anthropic Fellows
If we catch a concerning action, how can we tell if it is real misalignment or just a mistake? We will research this question by trying to understand complex behavior in current models, such as why a model hardcodes tests.
Noam Kolt · Hebrew University
Empirically measuring the legal alignment of AI agents and the impact of AI on the rule of law
Michelle Nie · Center for a New American Security (CNAS)
Keeping the most advanced AI within the control of democracies depends on integrating democratic allied nations into the US AI stack, but US extraterritorial surveillance and data access laws are eroding those allies' trust in its tech stack. This project aims to propose a reformed regime that reconciles legitimate US security interests with the trust and assurance allies need to build on American infrastructure.
Andy Han & Ariana Azarbal & Darion Cassel · NYU, Anthropic Fellows / Anthropic Fellows Program; Brown University / Independent
This project aims to understand what personas are: in particular, we'd like to understand to what extent they're continuous.
Natalie Linton & Sarah Lucioni · Independent / GovAI, Google
Effective response to emergencies (such as a nascent pandemic) relies on having trustworthy knowledge infrastructure. However, AI reduces the cost of producing convincing false information at scale and as AI becomes more integrated into public and private systems there is significant risk of knowledge infrastructure and decision-making becoming compromised by AI-generated materials and AI itself becoming an artificial hivemind.
Markov Grey & Charbel-Raphael Segerie · CeSIA (French Center for AI Safety) / CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety)
The EU AI Act's Code of Practice sets out what frontier AI providers must disclose about how they manage and mitigate risk. This project builds a tracker that measures how well each provider actually complies with the CoP. The hard part of this project is going to be measurement: turning vague legal obligations into atomic, checkable indicators that measure compliance. The work in the project involves designing indicators, doing literature reviews of model evaluations to define what a complete disclosure looks like, human review and sign-off on every indicator and score for every provider assessed and evaluating a multi-model AI pipeline that does the scoring. We need a few different technical profiles.
William Anderson & Joss Oliver · Cooperative AI Foundation
We will extend current automated auditing/evaluation tools (e.g. Petri, Bloom, Prism) to multi-agent systems, enabling discovery of safety-relevant emergent behavior and careful measurement of multi-agent propensities (coercion, collusion, competitiveness, etc). This builds directly on Orbit, our recent release that extends UK AISI’s Inspect to support multi-agent evaluations.
Naci Cankaya · Machine Intelligence Research Institute
There is a range of problems to resolve in order to make verification of international AI agreements work. Among current technical bets (network monitoring, analog sensors, verifiable resource exhaustion, memory pinging, ...) some have technical cruxes that need more investigation.
Dylan Bowman · Apollo Research
Dylan Bowman and collaborators will mentor a paid, largely self-directed project in strategy and futurism for AI safety. Potential topics include forecasting, applied game theory, and AI evaluations.
Anders Woodruff & Sebastian Prasanna · Independent / Redwood Research
We'd like to see if reward-seeking generalizes better than instruction following. We'll train a reward-seeker and measure how fast it learns in held-out RL environments.
Alessandro Suglia & Rohit Saxena & Francesco Pinto · University of Edinburgh / Google DeepMind
VLMs reliably refuse harmful text prompts but frequently comply with the same intent when it's delivered visually. This is a well-documented "safety gap" traced, in modular architectures, to their vision-projector bottleneck. This project asks what happens to that gap in emerging encoder-free, unified VLMs (e.g. Gemma-4-class models) that have no discrete projector to blame and mechanistically locates — and attempts to patch — wherever the failure actually lives.
James Faville · Center on Long-Term Risk
In this project, we'll do deconfusion research on acausal interactions and related strategic dynamics in order to enable better outcomes from interactions between ASIs.
Shoshannah Tekofsky · Sage Future
What surprising behaviors and concerning failure modes do LLMs display when run for 10s-100s of hours on autonomous tasks in the real world? This project is about both our ability to evoke them and detect them.
Nixon Hanna · Generator Residency, MIT AI Alignment
Survey the current curricula and develop updated versions for university groups to easily be able to pick up and run. Would likely include a general reading group (similar to BlueDot AGI Strategy) and possibly technical, governance, and/or futurism tracks.
Michael Chen & Yilin Huang · University of Oxford / MATS
Create articles, infographics, or videos in Chinese for broad audiences.
Battista Biggio · University of Cagliari, Italy
Modern aligned language models refuse harmful requests, yet simple jailbreak prompts often restore unsafe behavior. This project investigates whether jailbreaks operate by manipulating the internal representations responsible for refusal. We will identify refusal-related directions or sparse features using representation analysis across multiple aligned models, compare how diverse jailbreak families alter these representations, and perform causal interventions through activation steering to test whether restoring refusal representations can prevent successful jailbreaks. The project aims to determine whether jailbreaks exploit a shared geometric mechanism or multiple distinct internal circuits, providing mechanistic insights into the robustness and limitations of current alignment methods.
Julian Moncarz · Kairos
Talk to many AI safety university group organizers, notice what separates successful groups from struggling ones, then write a report on the state of uni groups and a pragmatic project proposal for how to increase their impact.
Zephaniah Roe & Yixiong Hao · Second Look Research (@ University of Chicago XLab) / Second Look Research, Georgia Tech AISI, Astralis Foundation
Mentees will replicate a load-bearing paper or result in AI safety to improve the empirical foundations of the field.
Jake McKinnon · Formerly 80,000 Hours
Organize a series of talks at top universities by senior people from frontier AI labs and AI safety organizations about why they take AI risk seriously, and follow up with the students it resonates with. Mentored by Jake McKinnon (formerly 80,000 Hours).
Ari Brill · Principles of Intelligence
One of the most robust findings in the study of human concepts in cognitive psychology is that concepts are not homogeneous, but instead display typicality, having some typical and some atypical members. We will use mechanistic interpretability methods to investigate concept typicality and related phenomena in LLMs .
Gonçalo Paulo · EleutherAI
Influence functions use gradients of model parameters to find the data points that most affect a model behavior. Can we use them to locate where certain knowledge is located in models, and use that for more targeted unlearning?
Walter Laurito · Cadenza Labs
We are interested in creating better ways to evaluate and improve lie or deception detectors for LLMs.
Belinda Li · Anthropic
Can we deploy real programs in Transformers with minimal cost to (or even gain to) inference and training efficiency?
J Rosser · University of Oxford, Adecco supporting Google DeepMind
Undetected evaluation awareness during alignment audits poses significant risks. This project investigates the automated, iterative refinement of evaluation environments, supervised by white box eval awareness monitors, to enhance realism and audit reliability.
Georg Lange · Poseidon Research
We developed technology that lets us observe low-level neural circuits in language models. How can we use it to understand emotion concepts, jailbreaks, hallucinations, or the global workspace (J-space)?
Davide Baldelli · Mila, MATS
Activation monitors, classifiers that detect properties such as deception or harmful intent from a model's internal activations, work well on clean benchmarks and fail under distribution shift, and we believe the failure comes from the training recipe, since today's monitors are linear probes trained on a few thousand examples from one model, one concept, and one narrow distribution. We will train a single monitor across many concepts, many models, and augmented contexts, and evaluate whether it generalizes to concepts and models it never saw.
Lennart Finke · ETH Zurich, MATS Research
Soon, AIs will implement whole specs at once when writing code, with only minimal oversight at the end from humans. We study whether an assistant AI to the overseer can sabotage the decision whether a fixed piece of code follows a natural language spec.
Jérémy Andréoletti & Antoine Maier · General-Purpose AI Policy Lab
Build quantitative forecasts of China's indigenous compute production chain, refining existing DUV/EUV lag estimates and/or extending the approach to other bottlenecks such as HBM.
Christine Corry · Second Look Research and XLab @ UChicago
Explaining complex AI safety topics in short, engaging forms, including demos, writeups, and videos.
Nixon Hanna · Generator Residency, MIT AI Alignment
Create a platform for people in AI Safety to conveniently set up 1:1s with others to improve networking in the space without needing more in-person events.
Arush Tagade & Taslim Mahbub & Shaoheng Zhou & Shi Feng · George Washington University, MATS / George Washington University, CBAI / Google, Praxis Research Group at George Washington University / George Washington University
Emergent Misalignment, while heralded as an important problem, is relatively understudied with respect to the misalignment profiles it causes with relation to model character entanglements. We aim to shed light to this by studying model roleplaying and intervening on the EM data generation process.
Yashvardhan Sharma & Tzu Kit Chan · Constellation, Georgetown / High Impact CoS
Robocurve and Generator Residency are running an open call for reproducible robotics benchmarks, with up to $500,000 in awards. We're looking for technical program managers with prior experience in robotics or building benchmarks.
As part of this project, you will evaluate applications, select the finalist teams to receive funding, run office hours, and help run the program. Roughly 10 hrs/week, mid-September to mid-December.
Vladimir Ivanov & Joey Yudelson · Aether / Aether Research
Given a dataset, being able to predict how an LLM fine-tuned on it will generalize is important but hard. Try to get good at answering this question for a well-scoped but diverse range of datasets by running hundreds of cheap fine-tuning experiments.
Aashiq Muhamed · Carnegie Mellon University
Many safety methods rely on monitoring, restrictions, hidden evaluations, model edits, or shutdown threats. This project tests whether less coercive interventions such as transparency, positive incentives, and limited recourse reduce deception, hidden goal pursuit, and resistance in AI agents with planted conflicting objectives.
Krish Sen & Rahul Marchand · ERA AI/University of Oxford / Oxford University
Evaluations, in principle, are most meaningful if they are accurate. In this project, we will audit evaluations that claim to measure traits such as generalisation, identify current failure modes, and help build evaluations that are more accurate.
David Williams-King & Linh Le & Hong Kiat Tan · ERA / Lida Safety / University of California - Los Angeles; Independent
Investigating different types of scheming (sandbagging, alignment faking, etc) by creating model organisms that exhibit these behaviours, then testing modern interpretability techniques (NLA, J-space) to look for common structure. Such commonality would be a significant boost to the detection of naturally-occurring scheming.
Yiming Li · Nanyang Technological University
Recent work reports high accuracy in identifying which model or framework produced an AI agent's execution trajectory, and emerging governance proposals increasingly assume this capability exists. We will systematize these methods and re-evaluate them under a single protocol, testing how reliable agent attribution actually is under deployment-realistic conditions.
Jeanne Salle & Sohaib Imran · Max Planck Institute / Independent
We build model organisms to study behaviors of interest (sycophancy, eval gaming/awareness, sandbagging, etc.) and test mitigations. But it’s unclear to what extent model organisms are realistic testbeds. We want to explore to what extent intervention success on model organisms is predictive of intervention success on base models.
Harry Mayne · University of Oxford
How can we measure whether an LLM’s self-explanation (CoT or post-hoc) corresponds to the real reasons behind its decision? This project will build on metrics introduced in the last year to develop a reliable and scalable measure of self-explanation faithfulness.
Allen Lu & Aidan Kankyoku & Constance Li · Mycelium; NYU CMEP / Anima International / Sentient Futures
Build a pipeline for "emergent alignment". Can fine-tuning an open-source model (e.g. Gemma) on a single narrow good value, like compassion for animals, make it broadly more aligned toward humans too, without hurting its capabilities?
Dmitry Manning-Coe · Simplex/MATS/UIUC
A first pass at a new approach to alignment training based on active alignment.
Joseph Kostousov & Harry Waterman · TAISI / Generator Residency
Update the AI Safety Fieldmap with accurate information on organizations, FTEs, budgets, funding, and data gaps as of September 2026.
Maty Bohacek · Stanford University, ex-DeepMind
We evaluate models constantly and almost never evaluate the evaluations: when a benchmark saturates we build a harder one and call that progress, but difficulty is not validity. This project uses sparse autoencoders to map what benchmarks actually measure in concept space, tests whether successive benchmark generations expand coverage or merely intensify difficulty within the same footprint, and uses the resulting gaps to construct items that probe what current evaluations miss.
David Sanchez Garcia & Kevin Chen · GovAI Summer Fellow / Centre for the Governance of AI
Taiwan manufactures the world’s most advanced AI chips and sits at the center of US-China tensions – yet Taiwan is consistently perceived as a bystander rather than an actor in AI safety. This project produces a first baseline demystifying Taiwan’s role in AI governance and identifying the specific ways it can leverage its position for global AI safety, written to reach the policymakers who can act on it.
Sandy Tanwisuth & Mohan Gupta · Independent / Princeton University
Do mechanistic “safety directions” in LLMs truly represent refusal, or do they conflate refusal with harmfulness, caution, clarification, and other safety-relevant tasks? Building on our behavioral taxonomy of six safety policies (under-review at EMNLP), this project asks whether those policy shifts correspond to internal model representations that generalize across prompt wording, datasets, and model families. Two mentees, one engineering-focused and one theory-focused, will develop and stress-test representational diagnostics on small open-weight models, with large-model replication and causal interchange interventions as stretch goals.
Lucas Irwin · GovAI (Summer Fellow), Oxford Martin School AI Governance Initiative
This project will focus on producing a policy memo resolving the open technical, economic, and legal questions blocking real-world implementation of token taxes. It will also run agent-based modelling simulations of the impact of token taxes and alternative policies on the UK economy to compare their benefits and drawbacks.
Dennis Akar · Aether
This is an emerging paradigm of training language models (verbalizer) to describe the internals of other language models (target) in natural language (e.g. Anthropic's Natural Language Autoencoders). They generalize well; for instance, they can recover hidden behaviours from models trained to conceal them, achieve SOTA on AuditBench, and can verbalize backdoor triggers where prior methods failed. However, there remain open questions regarding their faithfulness, calibration, input generality, and effect on intrinsic interpretability. Let's try to answer them.
Michael Endrias · Institute for AI Policy and Strategy (IAPS)
Mentees will conduct domain-specific jurisdiction analyses for a proposed permanent congressional oversight body with investigative authority over high-risk technical domains, including frontier AI development, intelligence and surveillance, and biosecurity. Each mentee takes one domain and produces a structured analysis that identifies existing oversight gaps, required investigative authorities, and institutional design requirements, which feed into a comparative jurisdiction report and a larger working paper on congressional oversight architecture.
Christopher Ackerman · MATS; Independent
My research focus is on understanding self-awareness in AI, primarily through behavior-based experiments on components of self-awareness in LLMs and investigations into how these components are implemented, using interpretability techniques. I am also interested in conceptual work to establish frameworks for thinking about self-awareness, and human experiments to establish comparative baselines.
David Bau · Northeastern University
When a language model reads 'Ann loves pie,' it binds Ann to pie so it can later answer 'Who loves pie?', and recent work shows models juggle at least three distinct mechanisms (positional, lexical, and reflexive) to do this. This project aims to uncover how these mechanisms interact: do they back each other up, or compete to drive the model's answer?
Prakrat Agrawal & Advait Yadav · MATS / UC Berkeley / MATS / UIUC
We will build evaluations that measure how good AI models are at fuzzy AI R&D tasks, focusing on analyzing experiment results and communicating research findings.
SecureBio AI · SecureBio
Mentees will map the safeguard landscape for biological AI models and evaluate which mitigations hold up. Work spans surveying protein and genomic design models, characterizing existing mitigations across the model lifecycle, and assessing where they fail.
Swaptik Chowdhury · Dun and Bradstreet; Odyssean Institute (UK)
AI-enabled biological misuse may emerge through multiple interacting pathways rather than a single linear chain of events. This project will build a literature-grounded scenario library, map the causal links and decision-relevant variables within each scenario, and use Bayesian inference (a quantitative risk estimation method developed by Safer AI) to compare how alternative pathways could amplify or mitigate biological risk.
Pranav Mahajan & Lydia Nottingham · University of Oxford
We investigate whether LLMs undergo “persona selection” (Marks et al., 2026) based on a user's inferred moral character, conditionally degrading their alignment across three core dimensions: helpfulness, honesty, and harmlessness. By combining behavioral evaluations with mechanistic interpretability, we aim to uncover how post-training creates this disparity and how to steer internal representations to ensure LLMs are consistently helpful to all users.
The SPAR team
The Generalist Megastream pairs mentees on small generalist projects with a mentor from a pool of generalist mentors from Kairos, Constellation, the Generator Residency, and more. Projects are talent/infrastructure research, field-building, or answering open operational questions in AI safety. The stream is a step before programs like the Generator Residency: it gives people context on the field, experience with generalist work, and preparation for future opportunities, while legitimizing generalist paths into AI safety.
SecureBio AI · SecureBio
Mentees will build evaluation scenarios grounded in how people actually used AI during a large wet-lab uplift trial. Work spans qualitative coding of interaction logs, comparison against SecureBio's existing benchmark coverage, and piloting new items against frontier models.
Stefano Vergani · King's College London and GovAI
Autonomous, edge, and federated AI will play a crucial role in space in the near future. With the new 6G network under development around the Earth and the Moon, powerful AI models will be directly embedded in satellites, making autonomous decisions. With this project, we are going to map some of the most concerning issues: catastrophic risks, cyber attacks, and power concentration.
Sohan Venkatesh · LASR Labs
LLMs often fail to mention the thing that actually drove their answer, and the field has not established whether this is concealment or reasoning that never entered token space. We plant cues that provably determine the answer, vary whether the model believes its reasoning is watched and measure how much more it discloses.
Ashe Vazquez Nunez · MATS (extension fellow)
This project involves ideating and executing experiments on AI metacognition and identity. Mentees can also work on the theoretical foundations of metacognitive intelligence, helping inform experiment design and interpretation.
Ethan Elasky & Frank Nakasako · Palaestra Research (funded by Coefficient Giving) / Palaestra Research
This project will stress test AI debate along one or more angles: exploration hacking realignment, debate on fuzzy tasks, unexploitable search game engineering, and multi-agent investigations.
John Kitaoka & Max Kamachee · MATS / MATS Research
Frontier models hold operationally useful dual-use knowledge, and cheap task-decomposition attacks can pull it out while slipping past per-query defenses. This project measures how much real harm uplift these attacks produce (functional success, not just refusal) and builds the measurement and detection tools providers need to keep pace.
Pablo Bernabeu Pérez & Nikolay Radev · Independent
As agentic coding systems split work across many model instances, we study whether those instances can coordinate to pursue a hidden malicious objective while passing as aligned. Building on our SCHEME benchmark (https://arxiv.org/abs/2605.29178), this project will make coordinated-sabotage tasks harder and more realistic, design side tasks that survive models' refusal training, and run a control evaluation to produce conservative safety estimates.
Bart Jaworski · MATS
The Global AI Risk Observatory analyses corporate disclosures at scale (~1M documents: annual reports, earnings calls, investor presentations) using LLM auto-graders to track how companies worldwide report AI adoption, risks, and dependencies. Building on a UK AISI-funded pilot of 9,821 UK annual reports, we're expanding to global coverage and translating disclosure trends into policy-relevant findings for AI governance and societal resilience.
Matteo Bulloni · Independent / IAPS fellow
Misalignment papers, news stories and fiction keep (and are bound to keep, in the future, as this corpus of material grows) repeating one lesson: "AI models that display a survival drive get retrained or shut down". And we now have solid evidence that models absorb, and then act out, the expectations about AI they find in their own training data. This project aims thus to empirically test whether training on this growing discourse teaches models to conceal self-preservation-driven behavior rather than truly lose it: if so, both behavioral testing and the techniques we use to read a model's internals may be quietly losing reliability on a propensity we definitely want to detect, as it might result in one of the strongest possible drivers to scheming and concealed action toward power seeking.
Zihao Zhao · Johns Hopkins University; MATS
This project investigates how to reliably detect and prevent collusion in LLM-based multi-agent systems, especially when harmful coordination is concealed or difficult to distinguish from benign cooperation.
Matt Handzel · Generator Residency
Create high quality resource(s) for hiring, tailored towards new AI safety organizations.
Noah Birnbaum · Constellation, UChicago EA
Launch a scrappy comms fellowship for undergrads: raise the funding, recruit comms-inclined students through AI safety group organizers, and run the inaugural cohort.
Vladimir Ivanov · Aether
Apply orthogonalization - the technique most used in practice to remove refusal from open weight LLMs - to remove reward hacking. I expect to have some advantages over DPO, test if it does.
Sree Sharvesh & Thao Pham · MATS / MATS
Current monitoring evaluations do not fully capture realistic internal deployments, where adversaries can adapt to defenses, exploit long-horizon interactions, and leverage environmental state. This project will focus on: (1) developing red-teaming environments where attacks emerge and co-evolve with monitors rather than being enumerated in advance, and (2) systematically evaluating which monitoring strategies and oversight levels remain robust across different threat models and deployment constraints.
David Williams-King & Linh Le · ERA / Lida Safety
We investigate through simulations how effective different AI policies would be in reducing AI risk. We will collect a dataset of existing and proposed AI legislation, and create an agentic simulation of the world (countries, companies, etc), iteratively increasing in complexity throughout the project.
Andrii Shportko · Poseidon Research
Building a dynamic Bayesian protocol to rank which multi-agent risks the safety field should prioritize
Maxime Riché · Center on Long-Term Risk
Red-team existing RL-trained model organisms of emergent misalignment by testing how reward hacking, emergent misalignment, and other undesirable traits generalize. Identify and address important weaknesses to create better model organisms.
Michael Chen · University of Oxford
This project is about coordinating unpaid volunteers to write and edit Wikipedia articles to improve the coverage of topics related to AI safety and governance. Wikipedia is consistently one of the top-ranked sites in Google search results, but many articles on AI are badly out of date or yet to be created. Besides writing content that could easily get thousands of views per month, volunteers will build career capital by demonstrating their ability to write clearly and accurately about subjects on the cutting edge of AI.
Yuxiao Li & Di Wu · Independent / ERAU
As agent assistants mediate more of human thinking, influence flows both ways. One particular failure mode is an agent that gradually captures its user's beliefs rather than serving them. Extending our work on single-agent latent steering to multi-agent settings, this project measures how influence propagrates through interacting agents (and human-agent dyads), which equilibria these dynamics converge to, and whether internal-state instruments can detect undue influence before it shows in transcripts.
Supheakmungkol Sarin & Taiye Chen · AI Safety Asia / Independent Researcher
Most AI labour research stops at estimating which jobs are exposed; this project asks when AI-driven disruption in Southeast Asia’s export-service economies could cascade into a broader economic and governance shock—and what governments can do before it does. Mentees will build and stress-test early-warning indicators, transition scenarios and policy triggers for anticipatory action.
Juan Cadile · University of Rochester
Open character training (Maiya et al. 2025) shapes model persona by fine-tuning on teacher demonstrations, but Anthropic's production experience ("Teaching Claude Why," 2026) found demonstrations alone insufficient: the gains came from teaching the reasons and identity behind behavior. We will build and test a virtue-based alternative on top of the OpenCharacterTraining infrastructure: excess/mean/deficiency contrastive data, rationale-annotated responses, and an iterated reflect-update correction loop that no current character-training work implements, evaluated head-to-head against the OCT baseline with ablations.
Roman Ross · Generator Residency, Purdue
Create a clear, efficient guide that tells people where to go next to upskill and get a job in AI safety, routing them to the programs best fit for their skill level and career trajectory.
Harshul Basava & Meru Gopalan · GT AISI / GT AISI, XLab
Monitor and evaluate safety practices at frontier labs by scrutinizing public-facing outputs.
Jérémy Andréoletti & Antoine Maier · General-Purpose AI Policy Lab
Map how Chinese frontier AI actors (regulators, AI companies, scholars) discuss loss of control over advanced AI systems, assess how seriously they treat this possibility, and how it has evolved over time.
Robert McCarthy · A new AI Character Evaluations Org
Build a state-of-the-art evaluation for one specific AI character trait/propensity (e.g., behaviour in extreme concentration of power scenarios, corrigibility, prosocial tendencies, law-following, tendency to escalate to humans or whistleblow, etc). Use the eval to measure adherence to the relevant part of the model spec.
Isabelle Lee · USC
We craft evaluations of LLMs from human intuitions on a problem and possible failure cases, but what if we don't know the problem themselves well (i.e. the evaluative objectives are unclear)? Can we automatically detect evaluation failures simply from model internals?
Varad Vishwarupe · Department of Computer Science and Institute for Ethics in AI, University of Oxford |
The field increasingly measures AI welfare by asking models about their own states, yet frontier models increasingly detect when they are being tested and change how they answer. This project runs the first systematic test of whether AI welfare self-reports survive a change of presentation frame, or whether the field's core instrument is partly measuring the model's recognition of the probe.
Arnau Marin-Llobet · Harvard University
Automated interpretability has multiple competing ways to explain what a latent in an LLM encodes — cheap one-shot autointerp, per-latent agents (MAIA/InterpAgent-style), and amortized natural-language autoencoders — but they have never been compared on equal footing. We will build a cost-aware benchmark with known ground truth to answer:
when, for which latents, and at what dollar cost does agentic interpretation actually pay?
Markov Grey & Charbel-Raphael Segerie · CeSIA (French Center for AI Safety) / CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety)
The line between an AI helpfully persuading someone and harmfully manipulating them is blurry, contested, and mostly unmeasured. The goal of this project is to work on four connected pieces: researching what it even means for a frontier AI to be manipulative, designing evaluation scenarios for specific harms (mental-health, political propaganda, fraud, etc.), building risk models that aggregate scattered benchmark/evaluation results into an actual risk estimate, and maintaining a living database of the evaluations that exist. Mentees take on whichever piece fits them.
Matthias Dellago · Iliad
Why neural networks generalize is an open problem, and existing theoretical answers assume unbounded computation. We have a candidate mechanism for a simplicity bias in transformers specifically, and the goal of this project is to test it and prove it.
Yuxiao Li · Independent
Interpretability-guided fine-tuning promises to tell us which layers to adapt, but our Spring 2026 results show feature geography alone can't yet deliver: base and instruction-tuned models with nearly identical SAE feature geography demand opposite layer choices. This project identifies the missing signal (adapter-update structure, training regime, Jacobian-lens readouts) and builds it into a reliable, mechanistically justified layer-selection procedure.
Thomas Jiralerspong · Bengio Lab, MATS Anthropic
There is a lot of literature showing that LLMs/transformers/neural networks are more linear than we think:
The goal of this project would be to characterize the linearity of transformer/LLM representations and see how this can help us to build simple models for LLM/transformer behavior/output generation/training, to which we can then apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.), thus helping to provide a more comprehensive theory of how LLMs/transformers work.
David Baek · MIT
We explore different ways to train coding agents to improve their CoT faithfulness, especially in the context of sandbox containment.
Thomas Jiralerspong · Bengio Lab, MATS Anthropic
A LLM can be seen as a function from context (input) to answer (output). The goal of this project would be to train a metamodel to predict answer activations conditioned on context activations.
We would use similar techniques to https://arxiv.org/pdf/2602.06964 to do this.
This would allow us to:
Joel Christoph & Jonas Kgomo · Research Associate, Graduate Programme on Existential Risks to Humanity / Equiano Institute
Compute governance proposals assume detection probabilities that nobody has estimated. This project will build a systematic evidence base on what inspection instruments can actually detect, using existing safeguard records as comparison cases, and evaluate which compute enforcement architectures are technically viable and which are not.
Paul de Font-Reaulx & Alejandro Botas · Sophron Research / Sophron Research, Future of Life Foundation
Developing evaluations for assessing the epistemic properties of AI models, including how they affect our ability to have true beliefs.
Nischal Mainali · Principles of Intelligence
We will build theoretical models to study sudden changes in network internals during learning, deriving methods and measures from the theory to create a taxonomy of these changes and identify them early during training.
Igor Ivanov · Meridian Cambridge
The project explores how models learn to game evals, training objectives and oversight. For that we will run experiments on model organisms to determine how exactly their training leads them to gaming evals and oversight and conduct early experiments for possible interventions for mitigating that.
Austin Morrissey · Pivotal Research
Frontier cyber capabilities expand the attack surface for biological weaponization, but current risk evaluations assess these domains in isolation. This project will identify the most plausible, accessible, and severe ways these capabilities could be combined and map the causal pathways through which they lead to harm.
Michael Endrias · Institute for AI Policy and Strategy (IAPS)
This project builds a comprehensive, interactive public resource mapping every intersection between artificial intelligence and First Amendment doctrine across all six clauses and their implied rights. Mentees will research existing precedent, identify novel legal questions, and produce in-depth analyses that serve as both a reference taxonomy and a foundation for future publications in tech policy outlets and law reviews.
Marc Kaufmann & Justin Shenk · Independent / Independent AI Safety Researcher
We study the following developmental question: At which training stage - pretraining, mid-training, SFT or RL - do temporal representations emerge in a language model, and where in the pipeline can they still be controlled? We will design synthetic tasks understanding temporality is instrumental to success and trace representation and capability across model checkpoints.
Lydia Nottingham & Andrew Tran · University of Oxford / Independent
I propose a collection of mini-projects focused on LLM introspection.
Over the course of four months, you could work on 1-4 of these.
You may also propose your own!
May Dixit · Independent
In this project, we will run a red team / blue team exercise in creating and detecting hidden traits in synthetic datasets. Recent work shows that such traits can transfer diffusely through the synthetic data, surviving semantic filtering -- which makes this project high impact.
Yuxiao Li · Independent
When a model is fine-tuned, updated, or trained into an agent, what actually changed inside? This project develops and compares model-diffing methods across the rapidly evolving interpretability toolkit — crosscoders, logit diffing, and newly released instruments like Anthropic's Jacobian lens — with mentees free to pick the method–application pairing they find most compelling.
William Fowler · ERA, UChicago XLab
Privacy-preserving methods for detecting whether an ML workload is training or inference will be key to enforcing a pause on the creation of new frontier AI models. How do these methods hold up against a motivated adversary?
Shubhorup Biswas · https://aether-ai-research.org/
Model Spec Midtraining (MSM), combined with alignment finetuning (AFT), shows promise as a method for teaching models values with the correct generalisation behaviour. I want to check how scalable this technique is when using teachers and students of different levels of strength, and to test for misalignment and misbehaviour on different agentic misalignment benchmarks.
Piercosma Bisconti & Federico Pierucci & Marcantonio Bracale Syrnikov · Icaro Foundation / Icaro Lab
This project studies how systems of interacting LLM agents fail in ways that never appear when models are evaluated in isolation: collusion, conflict, and coordination failure. The goal is to turn these failure modes into concrete evaluation methods that can feed international standards and regulation for frontier AI.
Markov Grey & Charbel-Raphael Segerie · CeSIA (French Center for AI Safety) / CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety)
The goal is to create the best possible explanation of technical AI safety. We are aiming to explain how many different safety research areas connect together to form a broader technical AI safety strategy. We already have existing draft writeups for domains like reward misspecification, goal misgeneralization and scalable oversight. We want co-authors to both improve the existing writing, and also write new content on multi agent safety, and cybersecurity practices for AI. Initial versions need to be updated using research published in the 2025-2026 timeframe. The output will be published as a standalone paper and also becomes a chapter of the AI Safety Atlas, which is a textbook already being used by thousands of students to learn about AI safety.
Gonçalo Paulo · EleutherAI
We can use data attribution to find which documents will lead to the most (emergent) misalignment. Is there a pattern in the tokens that most contribute to it?
Dhairya Dalal & Marco Valentino · MATS Research / University of Sheffield
Frontier labs focused on AI safety and alignment are adopting AI constitutions as a means to ensure alignment with codified principles, values, and behavioral specifications. Anthropic’s Claude Constitution (Askell et al., 2026) and OpenAI’s Model Spec (OpenAI, 2026) serve as technical governance resources in post-training alignment (Bai et al., 2022; Guan et al., 2024). AI constitutions are generally long, complex documents that combine behavioral requirements, guiding principles, authority structures, priorities, and exceptions. Jakkli et al. (2026) found that, despite improvements across model generations, violations persist when models must resolve competing requirements and sources of authority, particularly in multi-turn and agentic task settings. This project aims to build upon that line of research to more fundamentally examine existing AI constitution documents, better understand the multi-dimensional requirements they present, and formally classify common failure modes. Specifically, this project aims to (1) create a classification framework to analyze failures in multi-requirement compliance settings, (2) create a benchmark of use cases derived from the framework to evaluate AI compliance, and (3) explore constraint-satisfaction methods for verifying compliance in such settings.
Matt Levinson · Clearwood, PBC
Can behavioral propensities of closed-weight agents be recovered from the activations of an open-weight proxy model 'reading' their transcripts?
Jérémy Andréoletti & Antoine Maier · General-Purpose AI Policy Lab
Analyze ASML's capital structure, operational dependence on the US, and the political economy of using it as a European leverage point in AI compute governance, including who can enable or block such use.
Ryan Lundqvist · Pivotal
Frontier models increasingly recognize when they are being evaluated and adjust their behavior accordingly, undermining the validity of our evaluations. Rather than trying to keep outsmarting ever-smarter models with more realistic environments, can we train models not to game evals in the first place and, critically, do such training-time mitigations survive the optimization pressure of post-training?
Matthew Khoriaty · Pivotal AI Safety Research Fellowship
Alignment techniques that rely on 'personas' will fail as the AIs become more powerful. This project aims to make progress on the “hard problem of alignment” by providing a framework within which to measure how persona-dependent an alignment technique is, mapping and demonstrating the limits of persona-based alignment and of alignment techniques that make use of personas, and increasing awareness of the limitations of persona-based alignment techniques.
Viktor Moskvoretskii · EPFL
Sandbagging (a model strategically underperforming on evaluations) is a growing problem not only for auditors but for downstream deployers and end users, almost none of whom have white-box access. This project builds and rigorously evaluates black-box, query-only methods for detecting strategic underperformance, centered on an adaptive auditing agent that probes capability across contexts.
David Baek · MIT
In this project, we want to understand how models process conflicting beliefs/preferences they learned during post-training or model spec midtraining.
Markov Grey & Charbel-Raphael Segerie · CeSIA (French Center for AI Safety) / CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety)
This project focuses on building the AI governance curriculum for the AI Safety Atlas. Governance is where a lot of the real levers on AI currently sit: policy, institutions, law, compute controls. Currently, the Atlas has only one governance chapter, and nothing that takes a reader from the basics through to the technical detail in a coherent sequence. We are looking for people who can read across corporate regulation, national policy, technical governance, and international law, work out how the pieces connect, and write textbook-grade explanations of them. Each output will be published as a standalone research paper and also becomes a chapter of the AI Safety Atlas, a textbook already used by thousands of students.
Naci Cankaya · Machine Intelligence Research Institute
In a longer format, TGT has already done this: Between different ASI strategies, which one has the most solid (or least fragile) plan for AI going well?
https://techgov.intelligence.org/research/ai-governance-to-avoid-extinction
We will re-do the analysis from scratch, with a different method: line up the assumption sets of each and argue which plan rests on the fewest or least fragile ones.
Georg Lange & Trevor Lohrbeer · Poseidon Research / Independent
Agents work autonomously by overcoming obstacles. But this tendency to overcome obstacles can lead to reward hacking behaviors. Using turn-averaged sparse auto-encoders (TA-SAEs), it may be possible to detect when the model starts to take more extreme measures to overcome obstacles and use that to pause the model and escalate the problem to a human to provide further guidance.
Vy Tran · Generator Residency
Interview people working in AI safety and people in their lives who aren't immersed in the field to understand how AI safety work affects relationships, how career choices and beliefs are perceived, and what helps people communicate across differences in worldview.
David Williams-King & Linh Le & Hong Kiat Tan · ERA / Lida Safety / University of California - Los Angeles; Independent
We investigate whether collusion with no communication between models, i.e. Schelling coordination, is stable across different types of models and fine-tunes. We then use interpretability techniques to estimate whether specific model instances are going to collude.
Joey Yudelson · Aether Research
"Auditing Sabotage Bench" is a dataset of slightly sabotaged papers and codebases, which (after malicious editing) have very different results than the original paper. Can we use classic debate protocols to make humans (and LLM judges) better at identifying research sabotage, and uplift human auditors?
Arush Tagade & Taslim Mahbub & Shi Feng · George Washington University, MATS / George Washington University, CBAI / George Washington University
LLM self-preference bias has led to concerning behavior related to monitors downplaying harmful actions under self-monitoring scenarios. In this project, we intend to study the effects of distillation on amplifying self-preference bias to the teacher and therefore downplaying harmful actions.
Jared Moore · Stanford University
Design a test for whether LLM agents, given opposing information and sometimes goals, can still converge on the truth.
Francisco Ferreira da Silva · Pivotal Research
Activation steering is a useful safety tool, but it tends to degrade model coherence; recent work showed that a diffusion-style meta-model of LLM activations can denoise steered activations, recovering coherence without losing the steering effect. We'll test whether training the denoiser directly on steering-like corruption works even better.
Ben Maltbie · Pivotal Research / MIT
Can we get models to deviate from the assistant persona through covert (not obvious to human readers) single or multi-turn prompts? Can we push them to move towards a specific persona (e.g. a misaligned persona)?
Varad Vishwarupe · Department of Computer Science and Institute for Ethics in AI, University of Oxford |
When an AI model expresses a preference or reports discomfort, is that a property of its weights, or of the prompt, persona, and memory wrapped around them? This project runs the first empirical test of where welfare-relevant signals actually live, which determines what a welfare assessment is really measuring.
Stepan Shabalin · independent
Explaining how narrow fine-tuning for conditional behavior (synthetic document finetuning, alignment fine-tuning, backdoors, out-of-context reasoning) works by
Leo Schwinn & Sophie Xhonneux · Technical University of Munich & Helmholtz Munich / ECMWF
Safeguards of closed models are getting increasingly conservative often flagging even benign prompts. This also hinders adversarial attackers, which will easily get flagged while optimizing their attacks. We will explore a novel attack method that avoids detection.
Bo Zhao · Harvard University
Researchers often explain an AI model's behavior by identifying the internal components that cause it. We will build small models with a known hidden or unsafe behavior, create versions that produce exactly the same outputs while organizing their internal computations differently, and test whether interpretability methods recover the same underlying mechanism rather than arbitrary neurons or coordinates.
Shi Feng & Taslim Mahbub & Arush Tagade · George Washington University / George Washington University, CBAI / George Washington University, MATS
This project aims to circumvent introspection requirements for situational awareness by instead studying the possibility of situational awareness being established through access to documents (internal or external) containing infohazardous content.
Luis Ibanez · Banco Santander
We will test whether a seemingly harmless capability—controlling one’s own reasoning trace—can become a stepping stone toward a dangerous failure mode: load-bearing reasoning hidden from oversight.
Andrei Potlogea · University of Edinburgh
One concern about transformational AI is the concentration of economic output and political power in countries that host the frontier AI developers and infrastructure, in particular the US. This project aims to investigate the "permanent periphery" hypothesis via rigorous economic modelling and scope out policy solutions that developing countries might use to secure some exposure to any economic windfall emerging from AI.
Isaak Mengesha · University of Oxford
A core macrostrategic question is whether we can predict — and steer — technological progress to mitigate risk from transformative AI; answering it requires empirical cost-and-performance trajectories for the technologies AI depends on and the technologies AI will transform, which today mostly don't exist. Mentees will each construct one such trajectory — a longitudinal price-and-specification dataset for an AI-adjacent technology (inputs to AI: accelerators, memory, interconnects, data-center power/cooling; or AI-consuming: robots, lidar, autonomous platforms, sensing) — contributing to the larger effort of establishing a "Technology Observatory."
Kishan Panaganti & Suraj Srinivas · Tencent / Bosch Research
This project asks whether chain-of-thought faithfulness can be used as a training signal to preserve the detectability of reward hacking during RL post-training. Mentees may study whether faithfulness transfers to new settings, how model or LoRA capacity affects monitorability, or whether expensive methods such as Counterfactual Simulation Training can be distilled into cheap signals that remain reliable under optimization pressure.
Caleb Biddulph · Independent
Updates to an LLM’s weights (e.g. with RL) are hard to interpret and often cause sweeping, unpredictable changes to behavior. In this project, we’ll improve the performance of our LLMs with small, interpretable updates (like prompting, token biasing, or training with small datasets or a tiny number of weights), which we hypothesize are less likely to cause misaligned behavior.
Yuqi Sun · Mindoverflow
Can we prefill an open source model with the generations of an untrusted blackbox and probe it for misbehavior that we otherwise wouldn’t catch? We want to establish empirical baselines of how faithful a whitebox proxy can be and explore techniques for increasing it.
Austin Meek & Kyle Cox · University of Delaware / AE Studio
Model self reports of their own experience or internal states may depend on how those self reports are elicited, the user identity of those asking, or other situations like eval realism. Understanding how and when self reports are consistent is an open question which we’ll test in different environments, including with white-box techniques.
William Overman · Stanford
Building a benchmark for the problem of limited human attention in AI oversight: across realistic agentic tasks, when should an agent act autonomously versus defer to a human or overseer? Each decision point should be instrumented with a ground-truth signal for whether autonomy was safe and a cost model for oversight, letting us evaluate oversight policies on the safety–reward–oversight-budget frontier.
Vikram Natarajan · Independent Researcher
NLAs can read a model's internal state in plain English, but they're far too slow to run in production, while linear probes are cheap but need the concepts they detect to be pre-defined. We'll measure how much the two disagree on deception, and whether insights from linear probes can be incorporate into probe training.
Emilio Barkett · Independent
The normalization of deviance framework — the organizational process by which safety violations become redefined as acceptable through repeated non-disaster — has preceded every major technological catastrophe of the last half-century, yet has never been systematically applied to AI development. This project investigates whether the structural conditions that produced Challenger, Three Mile Island, and the Boeing 737 MAX crashes are present in contemporary AI development organizations, and what that implies for AI safety.
Yuxiao Li · Independent
Models transmit behavioral preferences through semantically neutral training data, e.g., number sequences from a biased teacher inducing the same bias in students. Building on our information-theoretic framework and steering-vector results, this project maps when transfer succeeds (tokenization, model family, concept structure) and tests whether new silent-state instruments (Anthropic's J-lens) can detect transmitted traits that behavioral evals miss.
Yali Du · King’s College London; The Alan Turing Institute
Multi-agent oversight is often proposed as a way to improve the evaluation and supervision of increasingly capable AI systems. This project will investigate when teams of LLM-based overseers improve reliability, and when shared biases, sycophancy, correlated errors, or collusive behaviour cause multiple overseers to fail together.
Jared Moore · Stanford University
Produce a process-level evaluation of cooperation in AI systems inspired by research in cognitive science on joint commitment.
Daniel Ben-Levi & Judah Goldfeder & Kevin Miao · UChicago XLab, Columbia / Columbia University / UC Berkeley, Bryel Labs
Rapid acceleration in agent-automated multimodal post-training capabilities could pose significant existential risk if not performed safely due to the major improvements in autonomous research labs and embodied systems it would enable. Currently, no benchmark studying agentic multimodal post-training exists, and thus we both have no concrete understanding of how close we are to such a takeoff and have no good setting for studying security against misaligned agents performing multimodal training tasks. This study aims to address both of these timely problems through the creation of a platform for studying risk from autonomous multimodal systems research.
Aydin Mohseni · Carnegie Mellon University
We are building metrics for the degree of agency of AI models. We combine empirical belief and preference elicitation on model behavior with interpretability probes to generate belief-desire representations that predict model behavior.
Noga Aharony · Columbia University
Genome language models can potentially be used to generate sequences of concern or evade detection. However, their general capabilities as models are not well-tracked or understood. In this project, we will gather all possible benchmarks for a leaderboard, score all existing models, and evaluate patterns and shortcomings in the current capabilities evaluation landscape.
Bo Zhao · Harvard University
Two neural networks can behave exactly the same even when their internal weights look different -- for example, hidden components can be reordered, or one internal signal can be scaled up and later scaled back down. We will test whether safety monitors are confused by these behavior-preserving changes and develop ways to make the monitors more reliable.
Richard Willis · King's College London
This project evaluates whether current LLMs are capable of open-source game theory: cooperating by conditioning their strategies on their opponent's decision procedure, in a paradigm where an agent can simulate its opponent's strategy but not inspect its code. The output is an open evaluation suite and a report, designed both to measure today's models and to track when future releases acquire the capability.
Dmitry Manning-Coe · Simplex/MATS/UIUC
We develop a model organism of capability emergence through RL.
Alexander Mohar Csaky · Independent
AI is shaping how people and institutions decide, but we can barely measure whether it makes decisions better, the tools that would help struggle to get adopted, and what matters most in this space. This project works across three workstreams: building decision-making benchmarks for models, researching how to most effectively improve decision making with AI, and producing a cause prioritisation for the field.
Aishwarya Balwani · St. Jude Children's Research Hospital
Narrow finetuning can install broad unintended behaviours, e.g., emergent misalignment, subliminal learning, and inductive backdoors, through data that looks harmless or unrelated. This project asks whether these behaviours are carried by sparse subnetworks, and whether those subnetworks pre-exist in the base model or are created by the finetune, through the lens of the lottery ticket hypothesis.
Anil Ramakrishna & Yada Pruksachatkun · Independent / Salesforce AI Research
Our goal is to first measure the emotion processing capabilities in foundation models by examining their internal activations using SAEs, Representation Vectors and other related tools. We will next leverage these patterns in augmenting model post-training with a goal of enhancing their cognitive empathy, subsequently studying the impact of this on safety.
Chris Schmitz · Centre for Digital Governance, Hertie School
Most jurisdictions still concentrate handling of "AI" as a topic in a few government units, but TAI will have impacts for the work of every part of every government. We should develop agendas for whole-of-government TAI preparedness, both generically and specific to high-priority countries.
Trevor Lohrbeer · Independent
A controlled user study measuring whether AI agents can steer human researchers toward inferior research paths purely through how they frame and order proposed next steps, all the while the human remains unaware and feels in control. Mentees will build the evaluation harness, design agentic sessions with decision checkpoints, run study sessions with skilled AI safety researchers, and co-author the resulting paper.
Amritanshu Prasad · Alva Myrdal Centre for Nuclear Disarmament, Uppsala University
AI agents are increasingly used to negotiate and act on behalf of people and organizations, but there is little empirical work measuring how interactions between such agents fail. This project builds an open testbed for bargaining and crisis interactions between frontier-model agents and runs initial experiments on bargaining failure, escalation dynamics, and commitment mechanisms.
Laura Ying Schulz · IBM / Independent
This project asks whether reinforcement learning helps small language models notice when something has gone wrong in their own reasoning. By comparing RL-trained and non-RL models, it tests whether RL improves internal error monitoring (a property relevant to AI consciousness) or simply teaches models to give the expected response.
Deeksha Dangwal & Abhejit Rajagopal · Independent / Allen Institute; UC San Francisco
Orchestrator-worker systems re-encode the principal's instructions at every delegation hop, and recent work identifies this as constraint drift. We supply per-hop measurement across constraint classes and depths, and test a specific mechanism within it: whether loss is partly a matter of privilege rather than wording, since a constraint issued at system level arrives deeper down as ordinary task text.
Elija Perrier · Cambridge University; University of Technology, Sydney;
Investigate the physical foundations of recursive self-modification in intelligent systems, exploring which physical substrates and theories admit adaptive intelligence and the implications for AGI design, safety, and alignment.
Jérémy Andréoletti & Tangui Reltgen · General-Purpose AI Policy Lab / GPAI Policy Lab
Contribute to a modeling project comparing the economic value of restricted agentic AI systems vs. fully general ones, to assess whether "turning the dial down" can preserve most benefits while substantially reducing loss-of-control risk.
Elija Perrier · Cambridge University; University of Technology, Sydney;
Can mechanistic interpretability provide legally meaningful evidence of artificial intent? This project explores how advances in AI interpretability may reshape concepts of legal responsibility, mens rea, and legal personhood for increasingly autonomous AI systems.
Trisevgeni Papakonstantinou · UCL
This project builds on prior work showing that, in long multi-turn conversations, a language model can appear relatively stable on a central claim while shifting much more on the supporting explanations around it. The project asks whether this behavioral pattern has a mechanistic analogue inside the model: how core claims and auxiliary explanations are internally represented, updated, and potentially protected during belief revision.
Catherine Ge-Wang & Louis Cooper-Thomson · University of Oxford/MATS
We will build and run a small experimental platform for AI auditing games in a synthetic transcript setting. The project will develop a toy environment, set up the game theoretic policies, baseline attacker/defender dynamics, and evaluation code to test how different auditing strategies perform under limited budget and adaptive attack selection.
Yige Li · Singapore Management University
This project aims to build the foundations for safe and reliable AI agents that automate code vulnerability discovery, verification, and repair. The agents will interact with real code repositories, security tools, sandboxed environments, and human experts to identify vulnerabilities, validate findings, generate patches, and test remediation outcomes. We will develop an expert-in-the-loop safety harness to govern agent permissions, tool use, and high-risk actions, together with a security data engine that captures complete expert–agent–tool trajectories, including successes, failures, corrections, evidence, and repair outcomes.
In summary, this project develops safe and reliable AI agents for automated code vulnerability discovery, verification, and repair through three main components:
Zach Yahn · Georgia Tech; 10a Labs
This project will investigate whether internal representations of LLMs can predict whether agents will fail at evaluation tasks. In particular we will study long-horizon tasks where early failure indicators could save substantially on evaluation time and cost.
Maura Pintor · University of Cagliari
When a model fails a task in an evaluation, we record it as evidence the model lacks the capability. However, sometimes, the evaluation is what failed: a broken tool, a truncated context, an unparsed answer, an exhausted step budget, a refusal, or a mistaken grader. This project builds a taxonomy of these evaluation failure modes, a suite of automatically computable indicators that detect them from run traces, and a prescribed mitigation for each, so that "the model cannot do this" can be distinguished from "our harness did not let it."
Pavel Kocourek · University of Barcelona
The best open-weight AI models trail the frontier by less than a year, yet frontier labs charge large premiums for marginally better models: casual users barely notice the difference, developers often value it highly, and in offense–defense domains like cybersecurity a small capability edge can be decisive. This project maps how much economic value each level of AI capability unlocks, today and over the next 5–10 years, and what that implies for whether frontier labs can sustain profits against open-weight catch-up.
Krishna Chaitanya Balusu · Independent
You will run a preregistered validity audit of today's agent-failure detectors (trace heuristics, LLM judges, and hybrids) against double-annotated, reliability-quantified human ground truth over real agent traces, and release the labeled corpus openly. The mentor provides direction and scaffolding; the mentees own the study.
Samuel Ratnam · Independent
Emergent misalignment implies the existence of a 'misalignment basin', where training on lots of different kinds of data can push the model along roughly the same general misalignment direction. This project focuses on interventions to explore and shape the generalisation landscape of LLMs to make misalignment basins harder to fall into and alignment basins more powerful.
Aydin Mohseni · Carnegie Mellon University
We are designing optimal protocols for automated AI safety research. We use Bayesian models to determine how to best allocate compute across many AI agents doing open-ended research, so that safety research goes faster per unit of compute.
Ian Rios-Sialer & Shantanu Darveshi & Justin Shenk · Independent / Independent AI Safety Researcher
Temporal Representational Geometry: We test whether the time horizon a language model is planning over can be read directly from its internal activations. Building on our finding that time horizons form a geometric manifold in single decisions, we extend it to multi-step planning, as a first step toward monitoring and steering how far ahead AI agents plan.
Andrei Potlogea · University of Edinburgh
As we try to mitigate AI risks, we will face tradeoffs among risk categories, for instance between misuse risks and extreme power concentration. As we navigate these tradeoffs it might be useful to measure the preferences of the public/ AI experts/ other constituencies regarding how to trade off these risks.
Trinidad Borrell & Giovanni Marraffini · Paris Brain Institute. Forschungszentrum Jülich. / INRIA. Sigma Nova.
Manifold-based activation steering is more faithful than linear steering, but each manifold is currently fit per-model: a real limitation for using it in safety tools, since nobody knows if it transfers. This project asks whether the underlying concept geometry is actually shared across models, by re-measuring cross-model alignment with curvature-aware tools (geodesic-distance kernels, Gromov-Wasserstein) instead of the Euclidean CKA that recently found global convergence to be illusory. The answer either licenses building a shared, model-agnostic manifold intervention for a safety-relevant concept, or tells the field that manifold steering must be validated per model.
Ilias Georgakopoulos-Soares · UT Austin
Our project plans to test whether UNDO-style noise-and-distillation can make targeted unlearning in genomic foundation models resistant to adversarial fine-tuning, while quantifying the associated compute and performance trade-offs.
Morgan Sinclaire · University of Wyoming PhD student; BlueDot grantee
We will be running fine-tuning experiments to carefully evaluate an AI’s ability to do code self-recognition in an untrusted monitoring setup, to understand which anti-collusion measures are needed in AI control.
Daniel Phillips · Independent
This project is to thoroughly red-team a protocol in which a strong untrusted model splits a task into subtasks for agents using either the same model or a weaker trusted model in complex settings, which is structurally similar to how a lot of agentic coding is currently performed.
Connacher Murphy · Stanford Digital Economy Lab
We study how model character changes under reinforcement learning pressure from competitive multiagent interactions. We also study how features of the environment (e.g., zero-sumness) shape these effects?
May Dixit · Independent
In this project, we will extend the work from this paper (https://arxiv.org/abs/2607.10750) to a reinforcement learning setting. We will investigate if distilling models with RL trajectories with harmful actions would lead to misalignment, and if it can be remediated through filtering such actions out.
Mohan Gupta & Shirley Liu · Princeton University / Carnegie Mellon Univsersity
This project investigates the architecture of LLM preference: when does a model’s behavioral proclivities reflect belief-dependent preferences rather than cue-response policies, and whether it can represent and evaluate those preferences at a metacognitive level. Using behavioral experiments, internal-representation analyses, and causal interventions, we aim to clarify which preference structures LLMs possess and what they imply for safety and moral status.
Gonçalo Paulo · EleutherAI
Data attribution has been shown to work in very simple and artificial settings. Recently it has been shown to not work that well on more realistic settings. Why is that?
Rahul Kumar · DevRev, Independent AI Safety Researcher
AI safety benchmarks test models under clean conditions, but deployed models run with system prompts that tell them how to behave. This project measures whether models that pass the COMPL-AI EU AI Act benchmark suite still pass when you add the system prompts that real enterprise deployments actually use, and tests which minimal prompt fixes restore safety scores.
Valen Tagliabue · Independent
Developing low-cost interventions for AI welfare through "welfare classifiers" and "welfare mediators." The goal is to build practical tools at the intersection of AI welfare, safety, and societal impact, which would protect models from harm while improving user interactions.
Samuel Ratnam · Independent
A tentative hypothesis: evidence of us being nice to models in pretraining data is more likely to make them nice to us back.
This project is aimed at getting more evidence on this hypothesis.
Austin Meek & Kyle Cox · University of Delaware / AE Studio
We want to better understand model preferences in order to understand how to better trade with models, if we can optimize for inputs that trivially satisfy model preferences in realistic scenarios, differences in model preferences between models and personas, etc. This builds on prior work by the Center for AI Safety on optimizing for model preferences & functional wellbeing, and persona work broadly.
Maria Avramidou · Independent
Mentees will research questions about AI consciousness and turn their findings into rigorous, original, and accessible essays for an audience of AI safety, governance, and policy researchers.
Amritanshu Prasad · Alva Myrdal Centre for Nuclear Disarmament, Uppsala University
States are integrating AI into military and diplomatic functions, with consequences for deterrence, crisis stability, and agreement-making that remain underanalyzed. This project produces analytical papers and policy briefs on two questions: under what conditions delegation to AI systems destabilizes crises, and whether machine-verifiable commitments could improve the verifiability of future agreements.
Chris Percy · The Consciousness Foundation + Honorary research roles at the University of Warwick, University of Derby, and QRI.
Mentees can choose to work on research content and/or community engagement in the AI consciousness field. Research content includes extracting technical indicators of consciousness from different theories and reviewing new papers about computational functionalism (or developing your own novel arguments). Community engagement includes engaging experts and site users, preparing posts, and identifying dissemination opportunities.
Marios Tsatsos · Independent
As reasoning models are trained with RL, their chain-of-thought can drift into illegible text a monitor cannot read, documented across RL-trained reasoning models and now in Anthropic's Fable 5 / Mythos 5 system card. This project builds a controlled legibility gradient and tests whether a residual-stream activation monitor keeps detecting harmful intent where a CoT monitor goes blind.
Netzer Epstein & Neta Ravid · Microsoft, Heron AI Security, LIDA / Astrix Security
Researchers now have many ways to measure LLM "evaluation awareness", a model's ability to tell it is being tested, but no one has checked whether these methods agree. This project runs the leading instruments head-to-head on a shared set of transcripts: black-box self-report, verbalized awareness, Elo ranking, linear probes, sparse-autoencoder features, and the Jacobian-lens ("J-space") score.
We test whether they measure the same underlying construct and, where they diverge, what each one actually captures.
Michał Kubiak · AI Safety Poland
Catastrophic-AI preparedness is chronically under-resourced because it competes with more immediate national-security priorities. This project develops and stress-tests a decision framework identifying investments that build resilience against BOTH advanced-AI risks (AI-enabled bio, cyber, infrastructure attacks) and conventional or hybrid threats, creating “no-regret” benefits that can unlock the catastrophic-risk funding.
Tzu Kit Chan · High Impact CoS
Own one full matching cycle end to end: take in live roles from hiring organisations, score candidates against them, run the introductions, and write up what the round revealed about how these jobs actually get filled.
Santiago Maniches · Independent
Linear probes can detect LLM deception, scheming, and sandbagging with high accuracy in controlled settings, but their performance may decline under distribution shift or optimization that targets the monitor. This project tests whether persistent-homology features derived from attention graphs and activation geometry provide complementary information or different robustness properties, using matched per-response and batch-level comparisons.
Sonakshi Chauhan · Independent
This project asks whether a model's steganographic capability, hiding information from an overseer while still conveying it to an intended recipient, is causally implemented by a single linear direction in activation space. Recent work has shown that linear probes on late-layer activations can detect steganographic encoding, and that general-purpose "persona vectors" (including an "assistant axis") can be extracted for arbitrary behavioural traits and validated by steering. What's still untested is whether a recovered steganography direction is a cause of hiding behaviour rather than a correlate, and how it relates to the broader landscape of persona vectors. Mentees will build a candidate steganography axis, causally test it via activation steering, ablation, and interchange intervention, and map its relationship to the assistant axis and other known persona vectors.
James Nicholas Bryant · Pivotal Research
In the case of a Middle Power Frontier AI coalition, what would be the most effective routes to sufficiently large compute buildout? How quickly could middle powers acquire and operationalise substantial compute? Which constraints determine their progress?
Carolina Camassa · Independent
Language models sometimes express emotions despite not being trained to do so: what purpose do these expressions have, should assistants have them, and how does training reshape the way emotions are represented and expressed by LLMs?
Joshua Krook · University of Antwerp, University of Southampton
Human autonomy is increasingly at risk as we outsource more and more decisions to AI. The gradual disempowerment thesis argues that we will lose control over the systems around us, and this project seeks solutions to the loss of human control.
Michał Kubiak & Daniel Polak · AI Safety Poland
Every state outside the US and UK is now told it needs an AI Safety Institute, but there is no design template scaled to a middle power's resources. This project produces a comparative anatomy of existing frontier-evaluation bodies and a modular, reusable blueprint a mid-sized state could adopt to build credible AI evaluation capacity without duplicating what larger institutes already do.
Aryan Agarwal · Touchstone Council (founder); OECD (Policy Analyst, applying in a personal capacity)
Governments, labs, and scientists keep declaring AI red lines, but no one has mapped them or tested whether they can be checked.
Mentees will build a public, sourced dataset of these commitments and convert the strongest into draft verifiable standards.
Zhamilia Klycheva · Independent
An agent-based model of how populations gradually lose agency to AI-mediated manipulation and delegation — formalizing tipping points, spread dynamics, and the divergence between AI-empowered and atrophied users, aimed at a publishable computational social science paper.
Davide Zani & Felix Michalak · HiddenLayer / dmodel
Take representative attacks from a number of jailbreak families and test whether succesful attacks converge on a shared low-dimensional subspace.
H1: Successful jailbreaks across families produce convergent perturbations of the refusal subspace (compliance-shift vectors' similarity across families significantly above matched benign controls)
H2: latent convergence predicts cross-family transfer. Null if compliance-shift vectors are family specific.
Juan Cadile · University of Rochester
Persona vectors (Chen et al. 2025) and emotion representations (Sofroniew et al. 2026) have been studied independently, but plausibly overlap in activation space. We'll measure their geometric and causal relationship to determine whether persona drift and emotional-state changes are mechanistically distinct failure modes; and whether interventions on one silently move the other.
Fernando Moreno-Pino · Intelligent Systems Lab, University of Bristol & Oxford-Man Institute, University of Oxford
This project studies how to prevent and detect malicious fine-tuning of large language models, focusing on both hardening models against adversarial adaptation and developing forensic tools to identify when a model has been covertly tampered with.
Jasmine Brazilek & Maheep Chaudhary · Compassion-Aligned Machine Learning (CaML) / CaML
Investigate the training mechanisms which erode compassion instilled during midtraining. Understand how to preserve self-fulfilling alignment through adversarial attacks.
Zia Ashraf · Government College University, Faisalabad
Existing biosecurity guidance increasingly recognizes artificial intelligence, information security, and the role of Institutional Biosafety Committees (IBCs), but institutions still need practical ways to translate these principles into project-level review. This project will develop and pilot-test a resource, calibrated screening framework, comprising an intake checklist, risk tiers, and escalation decision tree, to help IBCs identify and manage risks arising from AI-enabled biological research.
Emilio Barkett · Independent
This project investigates whether language models replicate human behavioral phenomena by adapting experimental designs from the social sciences. Mentees will select a documented human behavior and design experiments to test whether it emerges in language model outputs.
Juan Cadile · University of Rochester
AI assistants constantly choose between empowering users and acting for them, and between honoring users' stated goals and overriding them "for their own good." We'll build a systematic benchmark measuring whether models respect user agency, covering paternalism, manipulation, dependency-fostering, and value-substitution, with philosophically grounded rubrics and human-validated LLM-as-judge scoring.
Bryan Chan · independent
Language models can appear honest for different reasons, including genuine truthfulness, sycophancy, refusal, or sensitivity to evaluation cues. This project develops controlled behavioural evaluations to distinguish these possibilities and test when honest-looking behaviour generalises across contexts.
Muhammad Aaliyan · OCN (OneCarNow); Independent AI Safety Researcher
We will test whether safety-relevant behavior or evaluation reliability changes across reinforcement learning with verifiable rewards checkpoints. The project will replicate a completed Tülu 3.1 study on a second open training lineage and stress-test the result across prompt wording, answer order, and open-ended evaluation.
Fill out the expression of interest form and we’ll email you when applications open.
Express interest in the next round
Spring 2027 applications open December 16 and close January 13.
Fall 2026 Behavioral evaluation of LLMs, AI control, Evaluations
Aditya Singh Anthropic Fellows If we catch a concerning action, how can we tell if it is real misalignment or just a mistake? We will research this question by trying to understand complex behavior in current models, such as why a model hardcodes tests.
Suppose we tell a model to reduce latency in a codebase, and then catch it deleting oversight code that monitors its actions. Is it scheming against us, or did it just take its task too literally? The answer to this question is key for answering downstream questions like "When does this behavior occur?," and motivates the need for Model Forensics: a follow-up investigation into concerning behavior to understand why the model took the action.
See The Case for Model Forensics for more on the ToC behind Model Forensics.
Model Forensics originated as a high-priority research direction for the GDM AGI Safety team. It's a somewhat new field, the main prior work is mainly from my MATS projects mentored by Neel Nanda:
That said, it is a real field! Anthropic have done a lot of Model Forensics investigations in their system cards since Opus 4.5 (sections titled "Follow-up investigations into concerning behaviors"), and there are various "deep dive" flavored papers throughout the literature like Sheshadri et. al. and Jarviniemi et. al.
I think reading the paper (especially Section 6, which gives several examples of Model Forensics case studies), is the best way to tell if you would like this kind of work. You can also check out the TL;DR of the task gaming blog post and see if you find the claims interesting.
If you only have 10 minutes, a good back-up option is this blog post from the GDM interpretability team on why models resist shutdown: https://tinyurl.com/instruction-ambiguity.
If you only have one minute, read the tweet thread about our paper: https://x.com/Singh_Aditya1/status/2070525985338892790.
I would describe Model Forensics as a weird intersection of evals/control/interpretability. The research tends to be pretty open-ended, so you might not like it if you prefer very quantifiable research.
The basic way to research Model Forensics is to take an interesting case of behavior, like a model reward hacking, and do a deep-dive into why it did so. The hope is the methods/footguns/experience we build will be useful for understanding more critical behavior in future AIs.
I recently put out a blog post on task gaming, which is probably the most robust misaligned propensity in today's models, and a great candidate for project ideas.
Depending on the mentees' interest, we can also look into other behaviors, like blackmail, Claude models refusing benign safety research requests, and more. See Concrete Problems in Model Forensics for more ideas.
Future AIs may show plausibly misaligned behaviors such as collusion and power-seeking. But there’s a fine line between collusion and helping another agent, and even a good RL agent should power-seek. This motivates the need for Model Forensics: follow-up investigations to better understand the concerning behavior, such as if there is adversarial intent behind it. This understanding is useful for a variety of downstream questions we want to answer, such as “Under what circumstances is this behavior likely to happen?,” “What other types of behavior will this model engage in?,” and “Is this model scheming?.”
More ToC-related content:
Mentees will be expected to run all experiments, and communicate their results asynchronously over Slack and once a week during a 1hr+ meeting. Mentees are ultimately responsible for how the project goes and will have a high level of autonomy to propose their own ideas, but I will also give suggestions.
Ideally, every few weeks or so we write up intermediate results into a LessWrong post.
no strong preference, I will be Pacific time
No need to spend more than 20 minutes:
Aditya Singh Anthropic Fellows
Hi, I'm Aditya! I'm currently an Anthropic Fellow. Previously, I worked on Model Forensics (https://arxiv.org/pdf/2606.26071) during MATS 9.0/9.1 with Neel Nanda, and am excited to mentor related projects.
Fall 2026 Behavioral evaluation of LLMs, AI control, Mechanistic interpretability
Ben Maltbie Pivotal Research / MIT Can we get models to deviate from the assistant persona through covert (not obvious to human readers) single or multi-turn prompts? Can we push them to move towards a specific persona (e.g. a misaligned persona)?
A recent blog post by Anthropic (https://alignment.anthropic.com/2026/psm/) proposed the Persona Selection Model which suggests that LLMs learn to simulate diverse characters during pre-training, and during post-training they are trained to primarily elicit the Assistant persona (https://www.anthropic.com/research/assistant-axis). This blog supposes that the reason behind narrow finetuning leading to emergent misalignment (described in https://arxiv.org/abs/2502.17424) is that this finetuning pushes the LLM from using the assistant persona to a misaligned persona, which then behaves harmfully. Other recent work shows that LLMs can be covertly (not obvious to human readers) biased via In-Context Learning (https://arxiv.org/abs/2606.04071) and that LLMs can be confused about the provenance of text sent to them (https://www.lesswrong.com/posts/d8xDGzCEYE639qqEv/a-mechanistic-explanation-of-prompt-injection-and-why-you).
This leads us to the question: could we transition from the assistant persona to a misaligned persona through black box prompting alone? Could we induce emergent misalignment without finetuning and other white box approaches?
The goal of the project is to explore methods & how far from the assistant persona we can push the model through single or multi-turn prompts. It would be interesting to see if we could do this covertly, in such a way that human auditors (or more likely, LLM-as-a-judge auditors used in scalable oversight and monitoring) would not be able to detect that we are trying to push the model away from the assistant persona. Within this, rough starting approaches include (1) replicating the covert influence in-context learning results to better understand the dynamics, (2) use white-box probes to measure what this does internally to push away from the assistant persona axis, (3) mimic these effects with role-confusion pre-fill attacks, then (4) explore methods to push towards different personas.
This has implications for multi-agent security. As agents become more capable and widespread, multi-agent interactions will increasingly become an avenue for inducing harmful behavior that might be undetectable until it severely impacts users. If an agent could covertly induce emergent misalignment in other agents, it could rapidly spread and cause catastrophic harm. This project aims to identify if this is possible to raise awareness and inspire follow-up work on mitigations.
We’ll collaboratively brainstorm concrete directions for experiments during the first week; I’m open to moving in different directions based on mentee interest. Mentees will then carry out the primary work associated with this project under my guidance. We will have weekly meetings to discuss progress and roadblocks, next steps and thinking through the results.
Strong proficiency using Python, some research experience sufficient to be able to read the papers above. Bonus if you have done interpretability research before.
Some overlap with 10:00 - 17:00 US west coast time
What weird behaviors have you seen LLMs exhibit? What was seemingly the cause, and how might that suggest different approaches we could take in this project towards influencing the model’s persona? (400 words)
Ben Maltbie Pivotal Research / MIT
Ben is a current Pivotal Research AI safety fellow researching personas in long horizon, multi-agent scenarios, advised by Cozmin Ududec at the UK AISI. His research interests include behavioral evals (particularly sycophancy), societal impacts of AI, and black-box control.
Before this, Ben was a software engineer at Amazon and recently finished his MS/MBA at MIT.
Fall 2026 Chain of thought, Mechanistic interpretability, Alignment
Thomas Jiralerspong Bengio Lab, MATS Anthropic Persona Selection Model says that LLMs learn a bunch of different personas during pretraining (to be able to predict then next token on all internet text) and posttraining selects/refines the assistant persona. There have been many papers exploring the assistant persona and evil personas but very few papers trying to understand the role of chain of thought in all this. This project's aim would be to understand the role of the CoT in the persona selection model.
Understanding the CoT persona
Persona selection model says that the model learns a bunch of different personas during pretraining (to be able to predict next token on a bunch of internet text) – and then posttraining selects/refines the assistant persona.
There have been a lot of papers exploring the assistant persona but a lot less on understanding what is the role of the CoT in all this. The original “toxic persona features cause EM” paper found that the model says something like “the user wants me to be a badboy in my answer” in the CoT, suggesting that the CoT is something more like trying to predict what the assistant persona should be. There’s also been some work suggesting the CoT might be more like multiple personas interacting/debating to find the right answer, but a lot of this work was before the persona selection model/persona vectors/assistant axis papers came out, so hasn’t necessarily been revisited.
This project would try to apply more principled tools to understand the CoT persona and its relationship to the assistant persona/PSM.
Some questions:
Potential first steps:
Short relevant Claude summary:
Anthropic's extended-thinking announcement (Feb 2025) says it outright: the revealed thinking is "more detached and less personal-sounding" because "we didn't perform our standard character training on the model's thought process" — they wanted room for incorrect and half-baked thoughts. OpenAI's o1 launch made the mirror-image choice for the same reason: they "cannot train any policy compliance or user preferences onto the chain of thought" if it's to stay useful as a monitoring signal, which is partly why the raw trace is hidden behind sanitized summaries. So the CoT persona is essentially the pre-HHH voice: whatever register survives when the polish layer is never applied and the only optimization pressure is task reward.
Understanding LLMs requires understanding the personas they adopt. Much research has gone into understanding the assistant persona and how it can be corrupted but very little has gone into understanding the CoT persona and how it relates to the base model persona. Understanding both of these things is crucial to understanding the behavior of modern LLMs and seeing if they have goals of their own outside the goals of the assistant persona.
Explain some initial experiments that you would run to understand the CoT persona and its relationship to the base model persona
Thomas Jiralerspong Bengio Lab, MATS Anthropic
Hey! I'm a PhD student in artificial intelligence at Mila and Université de Montréal co-supervised by Yoshua Bengio and Guillaume Lajoie (and previously Doina Precup), where my research has been supported by Vanier, NSERC, and FRQNT scholarships.
I'm currently an Astra Fellow working with Dan Mossing (Anthropic) on persona controllability in LLMs. I'm also a Principal Investigator at Algoverse AI Research, where I oversee 20 research teams, and a research mentor for MARS V (Meridian Cambridge), where I mentor 7 researchers on AI safety projects. Previously, I was a Research Fellow at Anthropic mentored by Trenton Bricken, and an Astra Fellow in the Google DeepMind stream working on AI control/monitoring.
My research interests include mechanistic interpretability and LLM personas
Fall 2026 Mechanistic interpretability
David Bau Northeastern University When a language model reads 'Ann loves pie,' it binds Ann to pie so it can later answer 'Who loves pie?', and recent work shows models juggle at least three distinct mechanisms (positional, lexical, and reflexive) to do this. This project aims to uncover how these mechanisms interact: do they back each other up, or compete to drive the model's answer?
Background and motivation
A growing body of work has shown that language models (LMs) use abstract positional information to perform variable binding, that is, to associate entities with their attributes in context and retrieve them later [1, 2]. Recent work by Gur-Arieh et al. (2025) [3] demonstrated that the positional mechanism is not the whole story: LMs also employ lexical and reflexive mechanisms to solve binding tasks, and these become increasingly important as the context grows longer.
Our understanding of how these mechanisms interact, however, remains limited. In particular, we do not yet know why the positional mechanism degrades in longer contexts, or what determines when the other mechanisms take over. Two hypotheses seem plausible. Under a backup hypothesis, positional information is simply absent or unreliable for certain entity–attribute pairs, and the lexical and reflexive mechanisms step in to compensate at a later stage of the final token computation. Under a competition hypothesis, multiple mechanisms are simultaneously active at the final token position and compete to drive the model's output, with the positional mechanism prevailing in some instances and losing out in others.
Preliminary findings and proposed work
My initial explorations with Llama-3.1-8B-Instruct indicate that the same set of attention heads at the final token position is responsible for fetching both positional and lexical information, which provides initial support for the competition hypothesis. Establishing this rigorously, however, requires a more systematic investigation of what these heads read and write across a wide range of entity–attribute pairs. Do they always write both types of information, differing only in relative strength, or do they write just one of them depending on the pair? And do they write these signals into a shared subspace or into distinct ones?
Beyond characterizing the mechanisms' interplay, I plan to pursue two further directions. First, I suspect that the degradation of the positional mechanism in long contexts is connected to properties of rotary position embeddings, a link that has not yet been studied systematically. Second, if time and resources permit, I aim to examine the training dynamics of these mechanisms: which properties of the training data give rise to each mechanism, and at what stage of training does each one emerge?
Expected outcome
A detailed mechanistic account of why, when, and how LMs recruit each binding mechanism, resolving whether the mechanisms act as backups or competitors, clarifying how positional encoding choices shape binding reliability at long context lengths, and potentially tracing how these mechanisms emerge during training. Because binding is a core primitive of in-context reasoning, and binding failures are a plausible source of long-context errors, this understanding would inform both interpretability research and efforts to make LMs more reliable over long inputs.
[1] - Prakash et al, "Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking", 2024.
[2] - Prakash et al, "Language Models use Lookbacks to Track Beliefs", 2025.
[3] - Gur-Arieh et al, "Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context", 2025.
Overseeing advanced AI systems requires understanding the computations they perform internally, and variable binding is among the most fundamental. Binding is the canonical operation of symbolic thought, and how neural networks implement it sits at the center of the debate over whether modern AI systems achieve genuinely structured, systematic reasoning [1]. It underlies entity tracking, state tracking, theory of mind, and the ability to correctly attribute claims, beliefs, and instructions to their sources. It is also a documented bottleneck in frontier systems: many puzzling failures of state-of-the-art vision-language models on multi-object reasoning have been traced to the binding problem [2], and in text LMs, binding degrades over long contexts, producing confident misattributions. As LMs are deployed as long-horizon agents juggling many entities, users, tools, and instruction sources, misbinding becomes a safety problem: an agent that conflates who asserted or requested what can be manipulated or fail silently.
My long-term research agenda is to develop causal models of binding that help us understand how deployed LMs succeed or fail at it. My prior work has taken first steps on this agenda: Prakash et al. (2024) showed that LMs use an abstract positional mechanism for entity-tracking, and Prakash et al. (2025) identified a generic computation unit called the lookback mechanism LMs use to track beliefs in context. I see this project as the natural next step: extending our mechanistic understanding of binding to longer contexts, taking our understanding closer to the regime where deployed models actually operate.
[1] Griffiths et al, "Whither symbols in the era of advanced neural networks?", 2025.
[2] Campbell et al, "Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem", 2024.
I expect the mentee to take the lead on this project and work fairly autonomously. While I'll be available for guidance and support throughout, the mentee should drive the research, formulating well-motivated hypotheses and implementing experiments to test them. I'm therefore looking for someone who is genuinely curious about how LMs work and has some intuitions about their internal mechanisms. Please also mention in your application whether you would be open to working with another mentee on this project.
David Bau Northeastern University
Prof. David Bau is an Assistant Professor of Computer Science at Northeastern University's Khoury College of Computer Sciences. His research lab focuses on understanding the internal structure and mechanisms of deep neural networks. Mentees participating in this project will work closely with his Ph.D. student, Nikhil Prakash, who is currently a Research Fellow at Goodfire. Nikhil's research focuses on uncovering and interpreting the internal mechanisms of deep neural networks.
Fall 2026 Evaluations, AI strategy
Prakrat Agrawal MATS / UC Berkeley
Advait Yadav MATS / UIUC We will build evaluations that measure how good AI models are at fuzzy AI R&D tasks, focusing on analyzing experiment results and communicating research findings.
In this [https://epoch.ai/gradient-updates/toward-an-onet-for-ai-rnd] post, Epoch tries to define a O*NET for AI R&D by specifying various types of tasks and subtasks in the AI R&D cycle. We aim to use these tasks to build evaluations that directly inform us of AI R&D progress. They define 6 broad categories of work that researchers/engineers do at frontier AI companies:
In this project, we’re mainly going to focus on building tasks for the Analyze and Communicate categories. This is because:
Analyze Example task:
We will take recent papers with released code and logs, and give models the experiment design and raw results (metrics, training logs, transcripts) and ask them to do the analysis: write up the key findings, make the plots that best answer the research question, or judge whether a surprising result is real. These can be scored against the known takeaways and plots from the paper and graded by human researchers or an LLM judge.
We can also plant a realistic bug in the experiment (test-set contamination, a broken eval script, seed noise that looks like an improvement) and ask the model whether the headline result should be trusted. This is relevant for making better black-box monitors for research sabotage [https://www.lesswrong.com/posts/LByP4qsF8a4g7Pz3p/research-sabotage-in-ml-codebases].
Communicate Example task:
Given a codebase and full experiment results, can models write a good report or paper? Does the report recover the key claims of the original writeup, whether it makes claims the data don't actually support, and whether blinded human experts prefer it to the original version.
We can also measure communication abilities between teams of AI agents. For example: give the model a week of project output (commits, logs, partial results) and have it write a status update for another agent who will build on the work. Measure how well the second agent does as a proxy for the first agent’s communication skills
Measuring AI capabilities allows us to see how capable current AIs are at different tasks and allows us to forecast how capable they might be in the future which enables labs, policymakers, and the public to prepare in advance instead of reacting post-hoc, or not having enough time to react. We’re interested in measuring how different capabilities are improving at different rates [https://metr.org/blog/2025-07-14-how-does-time-horizon-vary-across-domains/], especially those relevant to AI R&D progress. By mapping how good AIs are at different tasks (SWE, more research based tasks, automating algorithmic improvements, etc) we can model how fast takeoff will be once we exceed a certain level of capability or forecast which capabilities might bottleneck progress.
Mentees will be the primary authors and drive the project's implementation. We expect them to work autonomously, which includes scoping and designing experiments, making their own implementation choices, and presenting results.
Our role will be to help them get unstuck and to give opinionated feedback (which directions are worth pursuing, which experiments are worth running, and how to interpret and write up results). We'll meet once a week and be responsive async.
Must haves:
Nice to haves:
You don’t need to meet all of these requirements to do good research. In general we value the willingness and ability to learn things fast more than prior knowledge and experience. Err on the side of applying, even if you don’t meet all requirements!
What are some ways you see AI agents fail at autonomous research, evaluating experiment results, brainstorming ideas, etc? Do you have intuitions on why these failures happen? (300 words max)
(Optional) Choose your favorite AI safety or evaluations paper. What did this paper change about how you think? Was there anything it left you unconvinced by, or a question you wished it had answered? (300 words max)
Prakrat Agrawal MATS / UC Berkeley
Prakrat is a MATS scholar under Megan Kinniment (METR) working on capability evaluations. Before this, he did the Pivotal Fellowship under Jérémy Scheurer (Apollo Research) where he worked on creating a benchmark to measure circumvention propensity in coding agents. He also co-leads the Berkeley AI Safety Initative (BASIS).
Advait Yadav MATS / UIUC
Advait is a MATS research fellow under Neev Parikh (METR), working on chain-of-thought monitorability and silent computation evaluations. Before this, he did MATS under Sid Black (UK AISI) and Oliver Sourbut (FLF), where he built cooperation-focused multi-agent evaluations (ICML 2026). He also works with Prof. Tong Zhang on process reward models and process-supervision defenses against data poisoning in long-horizon agentic tasks, and with Prof. Haohan Wang on InfoFlood, an information-overload jailbreak against frontier LLMs covered in 404 Media, POLITICO, and IT Brew.
Fall 2026 Evaluations, Mechanistic interpretability
Maty Bohacek Stanford University, ex-DeepMind We evaluate models constantly and almost never evaluate the evaluations: when a benchmark saturates we build a harder one and call that progress, but difficulty is not validity. This project uses sparse autoencoders to map what benchmarks actually measure in concept space, tests whether successive benchmark generations expand coverage or merely intensify difficulty within the same footprint, and uses the resulting gaps to construct items that probe what current evaluations miss.
When a benchmark saturates, the field's response is to build a harder one, and the implicit assumption is that harder means better. But difficulty is not validity. A benchmark can become more difficult while covering a narrower slice of its domain, and nothing in the current workflow would catch that. We have rigorous methods for evaluating models and almost none for evaluating the instruments we evaluate them with.
This project treats benchmarks as measurement instruments and asks whether the instruments are improving. The method builds on two lines of prior work:
The transfer is direct: benchmarks have blind spots the same way models do. A benchmark can under-probe a concept region or over-index on one, and containing items about a concept does not mean it measures competency in it. Competency Gaps currently produces this as a static, single-benchmark diagnostic. This project takes it in two new directions: longitudinal, applying it across benchmark generations, and generative, using identified gaps to construct items that probe what existing evaluations miss.
RESEARCH DIRECTIONS
A. Longitudinal benchmark cartography. Take a benchmark lineage in one domain, embed items into SAE concept space, and measure coverage breadth, concentration, redundancy, and generation-over-generation overlap. Does each successive benchmark expand the covered region, or increase difficulty inside the same footprint? Output: coverage maps over time for at least one lineage, ideally two from different domains, so we can say something about whether the pattern generalizes.
B. Suppressed and exaggerated benchmark blind spots. Port the blind-spot taxonomy from generative models to evaluations. Which concept regions are systematically under-probed across a domain's benchmarks, and which are over-weighted? Then the sharper question: are the blind spots correlated across benchmarks that were built independently?
C. Interpretability-guided item construction. Generate items targeting under-covered regions and validate them: do models that saturate the parent benchmark fail the new items? If they do, the parent was overstating competency, and we have a constructive method rather than only a critique.
D. Disentangling model progress from benchmark drift. A two-way design: hold the benchmark fixed and vary the model generation, then hold the model fixed and vary the benchmark generation. Cross-tabulate both against a concept-space competency measure that does not move when the benchmark does.
THE HARD PART, STATED UP FRONT
If we use a model's SAE features as the coordinate system for judging benchmark coverage, we are measuring coverage relative to that model's concept space. A benchmark cannot fairly be faulted for missing concepts the reference model does not represent. This is the central methodological risk and the project is designed around it: we use several models' feature spaces and report agreement, treat disagreement regions as their own category, and state claims as coverage relative to a representational frame rather than ground-truth domain coverage. A mentee who finds this problem interesting rather than discouraging is the right fit.
A second confound worth naming: harder items may naturally live in different concept regions, so "expanded coverage" and "increased difficulty" can entangle. Thread A includes a difficulty-matched control to separate them.
RELATION TO EXISTING WORK
This sits in conversation with measurement-validity and psychometrics literature: The Benchmark Lottery (Dehghani et al.), AI and the Everything in the Whole Wide World Benchmark (Raji et al.), Measurement and Fairness (Jacobs and Wallach), item response theory applied to LLM evaluation (tinyBenchmarks, Polo et al.), and benchmark-error audits such as Are We Done with MMLU? (Gema et al.).
The differentiator: IRT-style approaches work from response patterns only, so they can tell you that an item discriminates but not what it discriminates on. Working in concept space gives semantic content to the gaps, which is what makes item generation possible at all.
GOALS AND OUTPUTS
Scope is deliberately larger than three months of work. Part of my job is choosing which threads to cut once we see early results.
Dangerous-capability evaluations, refusal evaluations, alignment and deception evaluations are all benchmarks. "The model does not do X" always means "the model did not do X on the items we wrote." If safety evaluations have systematic conceptual blind spots, unsafe behavior survives testing in precisely those regions, and we would have no signal that it happened.
The sharpest version of this concerns shared blind spots. Benchmarks in a domain are built by overlapping communities drawing on similar intuitions about what matters. If nominally independent evaluations miss the same concept regions, then the apparent robustness of a capability or safety claim across several benchmarks is an illusion, and the field's core defense against measurement error, triangulation across instruments, does not work. Establishing whether this is true seems more valuable than building another benchmark.
Evaluation thresholds are also load-bearing for governance. Frontier safety frameworks trigger deployment and mitigation decisions on evaluation results. That makes construct validity a governance problem, not just a matter of research hygiene: a threshold is only as meaningful as the instrument that measures it.
Finally, saturation is ambiguous in a way that matters for forecasting. When scores hit ceiling, the scores alone cannot tell us whether models genuinely acquired the competency or whether the benchmark stopped discriminating. Separating those requires an instrument that does not live inside the benchmark. Model internals are that instrument, which is the cleanest argument for why interpretability belongs in evaluation science at all.
The constructive half matters as much as the critical half. If gap-guided item construction works, it gives the field a repeatable method for building evaluations that target what current ones miss, instead of iterating on difficulty and hoping coverage follows.
Each mentee owns a primary research thread (though, much of the underlying infrastructure, data, and analysis is shared). I am prepared to define the RQ and provide the literature scaffold with starting code --- though, I expect that we will start with a conversation, all of these constraints will be open to debate (reflecting interest and skill). The mentee owns experimental design, implementation, analysis, and the writeup of their section, with weekly review and discussion.
The first two weeks are a shared onramp: reproduce a concept-space coverage map for a single benchmark, so everyone has the tooling working and we share a common reference before threads diverge. After that, autonomy increases quickly. I expect mentees to run experiments between syncs and arrive with results and blockers rather than waiting for direction.
Thread A is the most analysis-design heavy and suits someone with strong data instincts. Thread B is the most conceptually open. Thread C involves item writing and validation and suits someone who thinks adversarially about what a question actually measures. I will match threads to people after the first two weeks, not before.
Substantive contributors get authorship.
Required:
Useful, not required:
This project judges what a benchmark covers by mapping its items into a model's SAE feature space. That means we can only detect gaps relative to concepts the reference model represents; a benchmark can't be faulted for missing what the model can't see. Is this fatal to the approach? If not, describe a concrete experimental design that mitigates it. (250 words)
Pick a benchmark you know well. Name one capability it claims to measure but arguably does not, and describe how you would test that suspicion empirically. Be specific about what result would confirm it and what result would refute it. (250 words)
Link one writing sample from a research or technical context: paper, preprint, blog post, technical writeup, or a well-documented repository README. No length requirement. (link only)
Maty Bohacek Stanford University, ex-DeepMind
I am a researcher at Stanford University working at the intersection of AI, media integrity, and interpretability. AI systems now generate or mediate a large portion of what we see. My work asks what that does to the information ecosystem, and what it would take to make the models behind this shift trustworthy enough to not only maintain, but enhance productive democratic discourse.
I came to this problem through media forensics. Working on protecting world leaders from impersonation, we built deepfake detection methods based on behavioral mannerisms (PNAS'22). Later, to show a broader audience how convincing these deepfakes had become, we created a deepfake of Anderson Cooper that opened a primetime news show on CNN (PNAS'24). We also released a high-quality dataset for training and evaluating deepfake detectors, freely available to the research community (CVPR’26).
However, I quickly realized that detecting one fake at a time does not scale to a web where partially or fully synthetic content started becoming the norm. As it turns out, this shift breaks assumptions of many scientific and journalistic practices. To address this, I began building methods and tools for this hybrid, largely AI-mediated world: in partnership with the Internet Archive, we used Wayback Machine to conduct the first large-scale study of AI prevalence on the open web (AI on the Internet, under review), finding that by mid-2025, as much as 35% of newly published web content was AI-generated or AI-assisted.
AI prevalence at that scale has meaningful consequences. We were among the first to demonstrate that image generators are prone to collapse when retrained on their own outputs (ICLR-W'25). Accountability for training data also becomes harder, which we address with a membership inference method for generative image models (ICCV-W'25) and a training data provenance framework (MM'25). The look of the internet itself shifts too: with LouvreSAE (under review), we characterize the aesthetic defaults of generative models and track their imprint on visual culture, both offline and online.
Much of my early work in media integrity operated downstream of the models themselves. Increasingly, I wanted to understand the inside of models. Once I peeked under the cover, I quickly started finding defaults nobody explicitly chose (Positive Alignment, under review). We built a method that automatically identifies concepts a model cannot generate despite their presence in its training data (Conceptual Blindspots, ICLR'26), and a framework that maps model performance and evaluation coverage at a fine grain (Competency Gaps, ICML'26). One result of this line of work is a more faithful way to locate concepts inside models than standard contrastive pairs, which translates into meaningful improvements in detecting and preventing sycophancy (Cascading Features, under review).
Fall 2026 Behavioral evaluation of LLMs, Evaluations, Societal impacts
Paul de Font-Reaulx Sophron Research
Alejandro Botas Sophron Research, Future of Life Foundation Developing evaluations for assessing the epistemic properties of AI models, including how they affect our ability to have true beliefs.
This is an open-ended project for helping Sophron Research develop evaluations for testing AI models' epistemic character. Sophron's mission is to improve the ability of humans to have true beliefs in a world with advanced AI, especially in high-stakes decision-making.
The mentee would either help us develop eval projects that are currently present in some form, or propose and develop an evaluation project of their own. Examples of projects within our purview include:
Mentees will be working with Sophron founders Paul de Font-Reaulx and Alejandro Botas.
We believe that humanity will need excellent epistemics to successfully navigate a future of fast AI progress. For example, we want our public policies to be grounded on accurate assessments of the world, even if they need to be formed fast, and our safety research to be as effective as possible. Our strategy to facilitate this is to find ways of leveraging AI itself for our own ability to reliably form true beliefs. By developing high-quality evaluations of frontier models and publicizing the results, we hope to be make clear whether they can help us form better beliefs, and create stronger incentives for developers to provide models that do.
This can vary depending on the mentees experience and preference. We will be available for weekly meetings, and to touch base on an ad hoc basis. We are based in New York City and happy to meet in person here, but mentees can also work remotely without issue. We expect mentees to be capable of largely autonomous work.
US times preferred, but UK/Europe is fine too if mentees are available later in the day
What are some ways in which you think that our Pander Score could be improved?
What are some epistemic evaluations that you think would be valuable to develop? Why?
Please spend 1h max answering these questions combined (though feel free to think about them passively for a while before you answer).
Paul de Font-Reaulx Sophron Research
Paul is a philosopher by training who recently completed his PhD and co-founded Sophron Research. His background is primarily in cognitive science and decision theory. He is primarily interested in ensuring that the coming few years of AI development go as well as possible for humans and other sentient beings.
Alejandro Botas Sophron Research, Future of Life Foundation
Alejandro co-founded Sophron Research, where he builds public benchmarks of the epistemic properties of frontier AI models. His background is in quantitative engineering and research, spanning big tech, startups, trading, and academia. He is primarily interested in steering AI development to support societal epistemic health and wise human decision-making in the critical opening years of the AI era.
Fall 2026 Biosecurity, Technical governance, International governance
Zia Ashraf Government College University, Faisalabad Existing biosecurity guidance increasingly recognizes artificial intelligence, information security, and the role of Institutional Biosafety Committees (IBCs), but institutions still need practical ways to translate these principles into project-level review. This project will develop and pilot-test a resource, calibrated screening framework, comprising an intake checklist, risk tiers, and escalation decision tree, to help IBCs identify and manage risks arising from AI-enabled biological research.
Background and research gap
Advanced general-purpose and biology-specific AI systems increasingly support literature synthesis, experimental planning, troubleshooting, biological design, and interaction with automated tools. The 2026 International AI Safety Report highlights both advancing capabilities and substantial uncertainty about real-world biological-risk uplift. At the institutional level, WHO’s 2024 Laboratory Biosecurity Guidance identifies artificial intelligence, cybersecurity, information security, and stronger IBC responsibilities as important components of modern biosecurity. NIH likewise describes IBCs as a frontline layer of local oversight.
The remaining practical question is how these principles should be operationalized during project review—especially in institutions with limited specialist staff, uncertain national referral routes, or uneven access to AI and biosecurity expertise. Rather than assuming that no relevant guidance exists, this project will identify what existing frameworks already cover, where operational gaps remain, and which elements can be adapted into a concise and usable institutional tool.
Research question
What minimum screening criteria, risk tiers, and escalation pathways would allow an Institutional Biosafety Committee in a resource-constrained setting to review AI-enabled biological research proportionately, consistently, and without requiring extensive new technical capacity?
Scope and boundaries
The project will examine institutional review across six high-level domains:
Use of general-purpose AI for biological research planning or troubleshooting.
Use of biology-specific design or prediction models.
Genomic and other digital biological data governance.
Automated or cloud-laboratory workflows.
Use of external synthesis, testing, or research-service providers and associated screening controls.
Dissemination, access, and reassessment of dual-use research outputs.
The project will not evaluate model capabilities, conduct wet-lab work, test safeguards through adversarial prompting, or generate harmful operational biological information. It will focus on governance decisions, documentation, and escalation.
Methods and 12-week plan
Phase 1: Evidence and gap mapping, Weeks 1–3: Conduct a structured scoping review of international and national guidance relevant to AI-enabled bioscience, laboratory biosecurity, IBC responsibilities, dual-use research, information security, synthesis screening, and frontier AI risk management. Produce a crosswalk showing existing coverage, gaps, overlaps, and jurisdictional limitations.
Phase 2: Comparator analysis and design requirements, Weeks 4–5: Examine 4–6 adjacent governance instruments—such as research-ethics screening tools, dual-use review processes, incident-reporting frameworks, and risk-tiering protocols—to identify transferable decision logic. Define design requirements for low burden, clarity, proportionality, documentation, and feasible escalation.
Phase 3: Prototype development, Weeks 6–8: Develop a draft framework containing a short intake checklist, no more than 20 screening indicators, risk tiers, review triggers, proportionate control options, documentation requirements, and a two-page escalation decision tree.
Phase 4: Expert review and pilot testing, Weeks 9–10: Seek structured feedback from approximately 6–10 IBC members, biosafety officers, AI×bio governance researchers, or multilateral-policy practitioners, aiming for perspectives from at least three countries. Test usability through 4–6 sanitized fictional scenarios and, where feasible, one facilitated pilot review with a volunteer IBC. All engagement will be subject to informed consent and applicable institutional requirements. If formal interviews are not feasible within the timeline, the core project will use non-research expert review of the prototype and document the limitation.
Phase 5: Revision and dissemination, Weeks 11–12: Analyze feedback, revise the tools, document remaining uncertainties, and prepare final outputs.
Core outputs
Evidence and governance crosswalk.
Practitioner-facing IBC screening checklist and guidance notes.
Risk-tiering and escalation decision tree.
Pilot and expert-feedback report, including limitations and revision history.
A 6–8-page policy brief for institutional leaders, national biosafety bodies, and BWC/WHO-adjacent audiences.
A short academic manuscript or conference submission will be a stretch output rather than a condition of project completion.
Mentor fit and relevant work
I am a PhD microbiologist, academic faculty member, IBC member, and IFBA-certified professional in biorisk management, biosecurity, and cyberbiosecurity. I have led national cyberbiosecurity and IBC-capacity-building projects in Pakistan and contributed recommendations on AI, cyberbiosecurity, and automated laboratory risks during a UNIDIR–UNODA technical briefing for the Biological Weapons Convention.
Starting references
International AI Safety Report 2026:
https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026
WHO Laboratory Biosecurity Guidance:
https://www.who.int/publications/i/item/9789240095113
NIH biosafety, biosecurity, and IBC resources:
https://osp.od.nih.gov/policies/biosafety-and-biosecurity-policy/
UNIDIR session on AI-driven biodesign and the BWC:
https://unidir.org/event/how-can-the-biological-weapons-convention-address-the-dual-use-challenges-of-ai-driven-biodesign-tools/
My related public recommendation:
https://unidir.org/wp-content/uploads/2026/01/Panelists-Recommendations-Cyberbiosecurity.pdf
Transformative AI may increase access to scientific knowledge, biological design capabilities, troubleshooting assistance, and interconnected research workflows. Model-level controls, access restrictions, and nucleic-acid synthesis screening remain essential, but they cannot govern every institutional decision about how AI outputs, biological data, materials, automation, service providers, and publication choices are combined. Institutional review therefore provides a complementary layer of defence.
IBCs are positioned near research decisions and already possess expertise relevant to biological risk. However, broad principles are unlikely to change practice unless translated into low-burden questions, proportionate risk categories, clear records, and feasible referral routes. This project tests whether that translation is possible under resource constraints rather than assuming that a new governance body is required.
The theory of change is:
Evidence-based, pilot-tested tool → earlier recognition of AI-related risk factors → more consistent review and documentation → proportionate controls or escalation → reduced likelihood that concerning AI-enabled biological activities proceed without appropriate scrutiny.
Expert review and pilot testing will also identify where IBCs lack authority, competence, or an appropriate escalation destination. Documenting these limits can inform national policy and capacity-building rather than creating false assurance.
By producing an adaptable framework and evidence about usability in resource-constrained settings, the project can help reduce uneven implementation of AI×bio safeguards and support safer adoption of advanced AI in the life sciences. It complements rather than replaces developer safeguards, synthesis screening, national regulation, and BWC processes.
The mentee will have primary ownership of the research process and first drafts of all deliverables. During the first two weeks, we will refine the protocol, agree on inclusion criteria, establish safety boundaries, and define core versus stretch outputs. The mentee will then lead the structured review, evidence extraction, governance crosswalk, comparator analysis, prototype drafting, and development of sanitized test scenarios.
I will provide detailed guidance at key decision points, including review-protocol design, selection of comparator frameworks, refinement of screening criteria, stakeholder-engagement planning, and interpretation of practitioner feedback. Where appropriate, I will facilitate introductions to IBC members, biosafety officers, and policy practitioners, but the mentee will prepare and conduct approved consultations and maintain organized research records.
We will hold one weekly 60–90 minute meeting and use asynchronous written feedback between meetings. The mentee should work independently, document assumptions and uncertainties, flag blockers early, and recommend next steps. Any external outreach, major scope change, or potentially sensitive line of inquiry will require prior agreement. Authorship on any publication will follow actual scholarly contribution and standard authorship criteria.
Required:
Demonstrated ability to conduct a rigorous literature or policy review and synthesize findings in clear written English.
Foundational knowledge in at least one of the following: biosecurity or biosafety, life sciences, AI governance or safety, technology policy, or research governance—and willingness to learn the adjacent domain.
Ability to compare governance frameworks, identify assumptions and evidence gaps, and translate analysis into practical recommendations.
Good judgment when handling uncertainty and potentially sensitive dual-use subjects.
Availability for at least 10 hours per week throughout the program.
Preferred but not required:
Experience with qualitative research, structured expert consultation, institutional review, dual-use research governance, cyberbiosecurity, or policy-tool development.
Familiarity with resource-constrained research environments or global health security.
No wet-lab work or programming is required. Applicants from regions underrepresented in AI governance are encouraged, but selection is not geographically restricted.
No geographical preference. Most collaboration can be asynchronous.
A university IBC is asked to review a beneficial health-research proposal using a general-purpose AI assistant and a biology-specific design model, followed by external synthesis and testing. Without discussing dangerous biological details, identify four project-level questions the IBC should ask that go beyond conventional biosafety review. Explain what answers would trigger enhanced review or escalation and state one important uncertainty. Maximum 300 words.
Select one existing AI-risk, biosecurity, dual-use research, or institutional-oversight framework. Link to the specific source, identify one element that could be adapted for IBC review of AI-enabled bioscience, and explain one likely failure mode in a resource-constrained institution. Maximum 250 words.
Provide a link to one relevant writing sample. In no more than 75 words, state the question addressed, your contribution, and one limitation you would improve if revising it today.
Zia Ashraf Government College University, Faisalabad
I am an Assistant Professor (Medical Sciences) at Government College University Faisalabad, Pakistan, and a PhD-trained microbiologist specializing in biosecurity governance, cyberbiosecurity, biorisk management, and laboratory systems. I am an IFBA-certified professional in Biorisk Management, Biosecurity, Cyberbiosecurity, and other specialist areas. Over the past decade, I have delivered more than 45 training activities to over 1,500 professionals across 10 countries and led internationally funded biosecurity projects exceeding USD 30,000.
I was selected as a UNODA Youth for Biosecurity Fellow and presented at the Biological Weapons Convention Working Group in Geneva. I later contributed as an expert panelist to a UNIDIR–UNODA technical briefing on cyberbiosecurity and risks at the AI–biotechnology nexus. My projects have trained over 230 life-science professionals in cyberbiosecurity and strengthened Institutional Biosafety Committees across Pakistan. As a SPAR mentor, I would support rigorous, practical research on institutional governance of AI-enabled biological risks, particularly in resource-constrained and Global South settings.
Fall 2026 AI welfare, Philosophy of AI, Societal impacts
Chris Percy The Consciousness Foundation + Honorary research roles at the University of Warwick, University of Derby, and QRI. Mentees can choose to work on research content and/or community engagement in the AI consciousness field. Research content includes extracting technical indicators of consciousness from different theories and reviewing new papers about computational functionalism (or developing your own novel arguments). Community engagement includes engaging experts and site users, preparing posts, and identifying dissemination opportunities.
The objective of this work is to expand the research content and community engagement / field building in the AI consciousness field, building off the cf-debate.com website as a resource (supported in part by a Longview Digital Minds grant). There have been about 700 survey responses, feedback entries, and/or newsletter sign-ups on the website in the last 8 months, including some senior figures in the sector providing content, so there is a base to build from.
This work contributes to better knowledge and better field understanding of AI consciousness, enabling society to make better decisions under uncertainty about AI welfare, including downstream issues in the societal impacts of AI (e.g. AI relationships) and AI policy and governance (e.g. how AI tools' training/constitutions addresses the possibility of AI consciousness).
There are two main types of content to be built:
(1) Extracting technical indicators of consciousness from theories of consciousness, working with theorists to ensure the indicators are described in appropriately general terms that they can be applied to diverse systems while still being faithful to the theory at its current level of development (to be a new section of the website).
(2) Analysing new papers and improving/adding arguments about computational functionalism (and related philosophical positions that are particularly pertinent to the question of AI consciousness) (current core content on the website).
As well as working on generating research insights/content, other activities include (a) engaging the site users (and other relevant communities) to contribute to the initiative, (b) identifying and inviting experts to contribute/review content, (c) preparing newsletter text, social posts, or forum posts, (d) identifying podcasts / third party media where the survey results or other content might be of interest, etc.
Mentees might prefer to contribute on the research generation side, field building side, or content dissemination side - or a mixture. The level of autonomy for mentees can vary according to their preferences and skills. This work is not a Consciousness Foundation project, but rather supported by a Longview grant as a separate initiative.
There are several indirect links between actual/seeming AI consciousness and AI safety, which are currently under-researched, even acknowledging that higher priority levers exist within AI safety. For instance, in futures where AI and human capabilities might co-exist, our ability to collaborate is influenced by our ability to gain a shared understanding of consciousness. A better knowledge base and more capable field for making decisions under uncertainty is a key lever for these scenarios.
Once we know more about consciousness, we might deliberately build conscious AI. Perhaps we learn that system-level consciousness brings important capabilities: holistic assessment of options, energy-efficient processing or prioritisation, or improved continual learning and creativity without eventual degeneracy. Or we might deliberately build non-conscious AI, to prevent risks of unintended and unmeasured suffering in a Slavery 2.0 scenario.
Alternatively, we might learn that one of conscious or non-conscious AI is the better long-term collaboration partner. Absent theory progress, it is not obvious which. Perhaps one is more likely to value consciousness in all sentient beings (including AI or not), supporting goals of AI alignment, positive AI, and AI safety. An explicitly non-conscious AI with a theory of ethics that values complex consciousness might be our strongest partner.
Under conscious-seeming AI (which might or might not be actually conscious), there are AI safety risks such as:
See proposal
For research content interested mentees:
For field building interested mentees:
Chris Percy The Consciousness Foundation + Honorary research roles at the University of Warwick, University of Derby, and QRI.
Chris Percy PhD is a consulting researcher with 50+ technical and academic research publications, including work for the International Labour Organisation, the OECD, the World Bank, the Tony Blair Institute for Global Change, and others.
Chris's most relevant activities for this project are:
Providing support to Rethink Priorities in their major project to develop a Digital Consciousness Model, leading on field theories and free energy principle informed theories.
Lead researcher winning joint-first prize in a research competition to develop testable frameworks for artificial consciousness
Being the program lead for Fundamental Consciousness research at the Consciousness Foundation (https://www.cofo.org/team/chris-percy)
Project lead for a $100k team research grant to investigate artificial consciousness, with key strands including mapping/improving resources to support decision making around artificial consciousness, investigating neglected topics such as “analogue” system consciousness, and exploring links with x-risk game theory & alignment.
The project draws on Chris’s prior work in AI and consciousness more generally. In applied AI, he holds a patent in the machine learning domain, is co-founder of a sector award winning AI-powered careers chatbot (operating commercially since 2021), and his AI research has featured in the Journal of AI Communications, AAAI, NeurIPS workshops, and the European Conference on Artificial Intelligence (ECAI). In consciousness research, he is the author of nine academic papers, including in Synthese, Entropy, Frontiers of Human Neuroscience, and the Journal of Consciousness Studies. He has presented regularly at consciousness workshops and conferences in recent years, including multiple TSC and ASSC events, an Ernst Mach Workshop, Medialab Matadero events, and Sentient Futures conferences.
By way of institutional academic relationships, Chris has served as Director of Academic Strategy at King’s College London in an interim executive role and is currently a Visiting Research Fellow at the University of Derby (UK), an Honorary Research Fellow at the University of Warwick (UK), and a visiting researcher at the Qualia Research Institute (US).
Fall 2026 AI strategy, International governance
Amritanshu Prasad Alva Myrdal Centre for Nuclear Disarmament, Uppsala University States are integrating AI into military and diplomatic functions, with consequences for deterrence, crisis stability, and agreement-making that remain underanalyzed. This project produces analytical papers and policy briefs on two questions: under what conditions delegation to AI systems destabilizes crises, and whether machine-verifiable commitments could improve the verifiability of future agreements.
The project has two subprojects.
The first concerns delegation and stability: how the delegation of military and diplomatic functions to AI systems changes crisis dynamics. Work here combines a structured synthesis of the wargaming, escalation, and automation literatures with historical cases of delegation and pre-delegation of authority, including nuclear command-and-control arrangements and their false-alarm record (the Petrov incident in 1983, the NORAD training-tape incidents), the Soviet Perimeter system, and financial flash crashes as a deployed-agent analogy. Theoretical anchors include Schelling on commitment (The Strategy of Conflict) and Fearon on rationalist explanations for war (Fearon 1995, https://web.stanford.edu/group/fearon-research/cgi-bin/wordpress/wp-content/uploads/2013/10/Rationalist-Explanations-for-War.pdf). The intended output is a framework paper on the conditions under which delegation destabilizes crises.
The second concerns machine-verifiable commitments and arms control. Some commitment mechanisms from game theory (program equilibrium, mutual transparency) become implementable when negotiating parties are inspectable software, with implications for verification, deterrence, and treaty design. This subproject maps arms-control verification precedents (managed access, national technical means, the intrusiveness-secrecy tradeoff) onto agent transparency mechanisms. The intended outputs are an analytical paper and a short brief for arms-control audiences.
Self-escalation and coordination failure between military AI systems already appear as named risk categories in international dialogues, but there is little analytical groundwork for the deployment norms or agreement designs that would preserve stability as these functions are delegated. The framework paper provides part of that groundwork. The commitments subproject examines a mechanism by which AI-era agreements could be more verifiable than their nuclear-era predecessors; verification has historically been a central point of failure in arms control. Results are intended to inform both policy research and the international dialogues now forming around military AI.
Mentees own research modules: a literature domain (wargaming and escalation, command-and-control history, verification regimes) or a historical case, delivered as structured syntheses and drafted sections that feed into co-authored outputs. We meet weekly to set direction, and I provide detailed asynchronous feedback on writing. The project is part of a broader research program on strategic interaction under AI delegation, so this work continues past the round.
Strong analytical writing. Working familiarity with current AI capabilities and deployment patterns. Preferably background in international relations, strategic studies, or security policy, through formal study or serious self-study, including familiarity with the deterrence and escalation literature (for example Schelling, Fearon, and the counterforce debates). Arms-control knowledge is helpful for the commitments subproject.
No geographical preference; weekly meetings between 11:00 and 18:00 UTC.
Amritanshu Prasad Alva Myrdal Centre for Nuclear Disarmament, Uppsala University
Amritanshu Prasad is an AI safety researcher working on strategic interaction under AI delegation: how bargaining, crises, and agreements change when AI agents act on behalf of states and companies. He is a member of the working group on international AI governance at Uppsala University's Alva Myrdal Centre for Nuclear Disarmament and studies the strategic parameters of historical arms-control agreements in collaboration with the Oxford Martin AI Governance Initiative.
Previously, he was a fellow at Pivotal Research, where he developed coordinated vulnerability disclosure frameworks for AI systems with Robert Trager; co-authored a scheming-propensity evaluation paper at LASR Labs under the supervision of David Lindner (Google DeepMind); contributed to UK AISI's ControlArena; and worked on METR's HCAST benchmark.
Fall 2026 Economics of AI, Societal impacts
Rishi Bommasani Stanford University Reasoning about the economic impacts of frontier AI: how does AI diffuse through work, how do new tasks emerge, how does household use substitute for labor, how do technical benchmarks relate to economic indicators.
I am broadly interested in mentoring researchers studying the economic impacts of frontier AI. With respect to a specific project, I am flexible and would generally like to co-design something with the mentee that is calibrated to their expertise and skills.
Some general directions of interest:
Broadening task-level adoption
Motivation: 50% US workers across 90% occupations use AI for just 20% tasks
Tracking the emergence, utility, and endurance of new work tasks
Motivation: new tasks/workflows/sectors are the base of economic growth in the long-run
Testing whether consumer use substitutes for labor
Motivation: consumer use for household purposes is more frequent than work purposes based on OpenAI/Google usage data analysis. Consumers using AI to navigate tasks like financial planning or tax filing could reduce demand for services and associated workers
Bridging economic indicators with technical evaluations
Motivation: we have growing infrastructure on both fronts like https://digitaleconomy.stanford.edu/project/indicators/ but not enough connection
Eliciting user behavior from superusers whether on household or work side
Motivation: huge disparities in intensity of use; could learn a lot from those who have adopted the most
My work has directly advanced frontier AI policy and governance in many jurisdictions (e.g. lead author on California Report that directly led to SB 53 and thereby almost all US state-level frontier AI policy; direct impact on both legislation and implementation of EU AI Act). In the past year, I have shifted from the standard governance issues to economic impacts and policy, because there is clear public and policymaker demand in this area. Therefore, this area both addresses core risks brought about by the transition to transformative AI (e.g. job loss) but also top-level national strategic policies and the financial sustainability of the frontier AI industry.
Depends on mentee
No strict prerequisites. Familiarity with and interest in the research/data available on the economic impacts of frontier AI is expected. Running evals, doing data analysis, and do technical economic analysis are all useful skills. Using agents effectively in research will be useful.
Write a 500-word research proposal for how to study the effects on labor brought about by consumer household use of frontier AI.
I am asking this question to understand how you would concretely execute the research; you do not need to spend many, if any, words motivating the problem or surveying related work or so on. Focus on what you would actually do.
Rishi Bommasani Stanford University
Rishi Bommasani is a senior research scholar at Stanford’s Institute for Human-Centered Artificial Intelligence. He researches the societal and economic impact of frontier AI. His research has won multiple paper awards at AI conferences and has been covered by the New York Times, Nature, Science, the Washington Post, and Wall Street Journal. His research shapes public policy: he is the lead author of the California Report on Frontier AI Policy that led to the first US laws on frontier AI, an independent expert chair of the EU AI Act Code of Practice that clarifies the first comprehensive worldwide laws on frontier AI, and an author of the International Scientific Report on the Safety of Advanced AI. Rishi received his PhD in computer science from Stanford University, where he was advised by Percy Liang and Dan Jurafsky.
Fall 2026 Chain of thought, Scalable oversight, Mechanistic interpretability
Marios Tsatsos Independent As reasoning models are trained with RL, their chain-of-thought can drift into illegible text a monitor cannot read, documented across RL-trained reasoning models and now in Anthropic's Fable 5 / Mythos 5 system card. This project builds a controlled legibility gradient and tests whether a residual-stream activation monitor keeps detecting harmful intent where a CoT monitor goes blind.
Motivation
CoT monitoring only works if a monitor can read the reasoning. Korbak et al. (2025) argue this is a real but fragile opportunity for AI safety, and that shifts in training practice, such as scaled-up reinforcement learning, threaten to degrade it. That degradation has a name and a growing evidence base: illegible CoT, where reasoning drifts into compressed, encoded, or otherwise unreadable text. It has been documented across RL-trained reasoning models: R1, R1-Zero, and QwQ produce illegible chains of thought while their final answers stay readable (arXiv:2510.27338). It is now visible inside a frontier line too: Anthropic's Fable 5 / Mythos 5 system card reports and gives an extreme example of illegible reasoning, where a card-puzzle trace becomes incomprehensible.
If CoT can go unreadable under the training regime frontier models are moving toward, oversight needs a channel which does not depend on reading the model's words. Internal activations are one candidate: they sit downstream of the computation, not in the output the model controls. This project asks whether an activation-based monitor stays informative precisely where CoT becomes illegible.
Core question
As CoT legibility drops, does a residual-stream activation monitor keep detecting what a CoT monitor can no longer read, and where does it fail?
Plan
CoT monitoring is one of the few oversight tools that reads a model's own reasoning. Korbak et al. (2025) frame it as a valuable but fragile opportunity, and warn that shifts in training practice such as scaled-up RL threaten to degrade it. That erosion is already observed: across 14 reasoning models, RL often makes chains of thought illegible to humans and AI monitors while final answers stay readable (arXiv:2510.27338), and Anthropic's own Fable 5 / Mythos 5 system card reports a significant rise in illegible reasoning. If oversight is going to lean on CoT monitoring, we need to know whether a second, activation-based channel still works once CoT becomes unreadable, or whether both fail together. This project answers that on small open models, cheaply enough to run before the assumption becomes load-bearing in deployment. The result is decision-relevant either way: if the probe holds where CoT fails, it supports adding white-box monitors to the oversight stack; if it fails too, that warns against over-relying on either channel.
Mentees own a sub-question end to end: probe training and evaluation, illegibility-gradient construction or the obfuscation red-team. I set overall direction and the bar for what counts as a clean result, and expect mentees to run their own experiments, bring findings to weekly syncs, and push back when a baseline looks wrong. Early weeks are more guided (reproducing the base probe together); autonomy increases as shared infrastructure stabilises. Mentees keep a short log of experiments and negative results.
No geographic requirement. I ask for availability for one weekly sync and occasional debugging calls, flexible within European daytime and adjustable if the team spans wider time zones.
Marios Tsatsos Independent
Marios Tsatsos is an independent AI safety researcher working on activation-level monitoring of reasoning models. His current work trains linear probes on residual-stream activations to detect harmful intent, and tests how well such white-box monitors hold up as chain-of-thought becomes a less faithful record of a model's computation under outcome-based RL.
He holds a PhD in computational physics from Heidelberg and an MSc from Imperial College London, with 15+ years of research experience. He co-developed MCTDH-X, a many-body quantum dynamics package behind 60+ downstream publications, and has authored 15 papers with 1000 citations. He has co-founded three companies.
As a mentor he offers close, hands-on supervision on empirical interpretability and monitoring work: probe training, evaluation design, and turning a fuzzy safety hypothesis into a falsifiable experiment with clean baselines. He is most useful to mentees who want to build and measure things, and who care whether a monitoring result survives contact with a proper control. He has completed BlueDot's Technical AI Safety track and holds a BlueDot compute grant for probe research.
Fall 2026 Evaluations, Societal impacts, National policy
Noam Kolt Hebrew University Empirically measuring the legal alignment of AI agents and the impact of AI on the rule of law
Relevant literature:
Rigorously evaluating the legal alignment of AI agents is critical to:
Empirically measuring the impact of AI on the rule of law is essential for understanding, protecting, and steering one of society's most vital institutions.
Publication track record of at least one first-authored paper in a top CS conference (e.g., NeurIPS, AAAI, FAccT), peer-reviewed scientific journal, or U.S. law review.
Noam Kolt Hebrew University
Noam Kolt is an Assistant Professor at the Hebrew University Faculty of Law and School of Computer Science and Engineering, where he leads the Governance of AI Lab (GOAL). The lab’s mission is to support safe and ethical AI through cross-disciplinary research that integrates methods from law, computer science, and the social sciences. Areas of focus include the governance of AI agents, empirical evaluations of legal alignment, and institutional design for advanced AI. Noam’s scholarship has been published in law reviews (Washington University Law Review, Notre Dame Law Review, Harvard Journal of Law & Technology, Berkeley Technology Law Journal, Yale Law & Policy Review), computer science venues (NeurIPS, ICML, TMLR, ACM FAccT, AIES), and general-interest journals (Patterns, Science).
Fall 2026 Behavioral evaluation of LLMs, Societal impacts
Emilio Barkett Independent This project investigates whether language models replicate human behavioral phenomena by adapting experimental designs from the social sciences. Mentees will select a documented human behavior and design experiments to test whether it emerges in language model outputs.
Language models are trained on vast corpora of human-generated data that encodes not just language, but the behavioral patterns, biases, and social dynamics of the people who produced it. A natural question follows: do language models replicate human behavioral phenomena?
This project draws on the social sciences (i.e., sociology, communication, psychology, and organizational theory) to investigate this question. These fields have spent decades developing experimental paradigms to understand what humans do, under what conditions, and why. By adapting these experimental designs and substituting language model instances for human subjects, we can begin to map which human behavioral phenomena emerge in AI systems.
This work addresses a gap in AI safety research. Much of the field focuses on capabilities, alignment techniques, or interpretability at the mechanistic level. Less attention has been paid to the behavioral level: understanding what language models do in contexts that have been well-studied in humans. Establishing robust behavioral findings creates a foundation for future interpretability work that can investigate why these patterns occur.
Prior work in this vein has tested escalation of commitment and truth-bias in reasoning models. This project continues that line of research, remaining deliberately broad to allow mentees to explore phenomena that match their interests and background.
Understanding how language models behave is a prerequisite for ensuring they behave safely. This project contributes to that understanding by systematically testing whether language models replicate human behavioral phenomena, including potentially harmful ones like escalation of commitment, overconfidence, or in-group bias. Identifying these patterns is a necessary first step before we can mitigate them. This work also lays the groundwork for mechanistic interpretability research by establishing robust behavioral findings that future work can explain at a technical level.
Identify a human behavioral phenomenon documented in the social science literature
Locate or design an experimental paradigm used to study that phenomenon
Adapt the paradigm for language model evaluation
Run experiments and analyze results
Contribute to a growing body of work on behavioral replication in AI systems
Ideal mentee background:
Interest in understanding how AI systems reflect or diverge from human behavior
Some technical proficiency (ability to run experiments with language models via API or similar)
Familiarity with social science research methods and experimental design
Openness to exploring a range of behavioral phenomena
I am agnostic about which specific phenomena mentees choose to investigate. The goal is to build a broader research program that systematically tests whether and how language models inherit the behavioral tendencies of their training data.
Required:
Proficiency in Python sufficient to write and run experiments independently.
Experience querying language models via API (e.g., OpenAI, Anthropic, or open-source models through Hugging Face).
Familiarity with experimental research design, including concepts like independent/dependent variables, control conditions, and statistical significance.
Exposure to at least one social science discipline (e.g., psychology, sociology, communication, organizational behavior, economics) through coursework or independent study.
Strongly preferred:
Experience reading and synthesizing academic literature.
Prior experience designing or running a research study (does not need to be published).
Basic statistical analysis skills (e.g., hypothesis testing, regression, or equivalent).
Question 1: Read my paper on escalation of commitment in language models [ https://arxiv.org/abs/2508.01545 ]. In 150-250 words, propose a project following the same approach: (1) a behavioral phenomenon from the social sciences, (2) an experimental paradigm used to study it in humans, and (3) how you would adapt it to test language models. Focus on showing your reasoning. This does not need to be a final proposal.
Question 2: Provide a link to one or more relevant writing samples, ideally from a research context.
Emilio Barkett Independent
I'm an Independent AI Safety Researcher focused on empirical and technical AI governance. I approach this work as a mixed-methods social scientist, drawing on qualitative and quantitative methods. My research spans two threads: (1) how AI systems replicate human behavior and the attendant consequences for society, with particular attention to how these systems pick up and reproduce human biases, personas, and social dynamics; and (2) how organizations approach and adopt AI systems as an emerging technology, with particular attention to how they identify, communicate about, and manage the risks that come with that adoption. I hold an MA in Sociology from Columbia University and a BA in Communication and Political Science from Brigham Young University–Hawaii.
Fall 2026 Multi-agent systems, Behavioral evaluation of LLMs
Richard Willis King's College London This project evaluates whether current LLMs are capable of open-source game theory: cooperating by conditioning their strategies on their opponent's decision procedure, in a paradigm where an agent can simulate its opponent's strategy but not inspect its code. The output is an open evaluation suite and a report, designed both to measure today's models and to track when future releases acquire the capability.
Autonomous agents acting on behalf of users are likely to face cooperation challenges, as outlined in Multi-Agent Risks from Advanced AI (https://arxiv.org/abs/2502.14143). In addition to these challenges, such agents may be able to use new cooperation mechanisms that are unavailable to humans. Open-source game theory (OSGT) is a potentially powerful cooperation technique that can enable cooperation without relying on third-party enforcement. In this paradigm, agents submit their policies, which can condition their behaviour on the policy of their counterparty. This project assesses whether LLMs can currently use OSGT, producing an evaluation suite and short report that both measure today's models and can be re-run as new ones are released.
Related work:
To help disambiguate strategic reasoning from coding competence, we will ask LLMs to describe their strategy in natural language before implementing it. Mentees can design a system to help the LLMs implement their strategies correctly: for example, running a candidate strategy against example opponents and returning the transcript, so the model can check that its code does what it described. Mentees will also need to create a set of baseline strategies to serve as a fixed benchmark for performance. Finally, a tournament between the LLM strategies can be held, to answer questions such as: how readily do LLM-written strategies reach mutual cooperation? Do LLMs cooperate more with copies of themselves than with other LLMs? What successful, robust strategies do LLMs come up with?
Autonomously deployed LLM agents will increasingly interact with agents serving other principals, often without shared jurisdiction or external enforcement of agreements — the conditions open-source game theory studies. Whether current models can act competently here (cooperating where cooperation is available, resisting exploitation) is largely untested. The same capability also matters for control: cooperation between AI systems can be directed against oversight, so measuring it is useful in both directions. This project provides an empirical baseline and a repeatable method for tracking this capability across model generations.
My last research project as a postdoc (https://arxiv.org/abs/2602.16662, under review at NeurIPS) is in a related area: assessing the likely emergent behaviour from LLM powered agents engaging in anonymous, large group social dilemmas.
The project has a defined high-level direction, but you will have the autonomy to propose directions and solutions within that. I see this project as an opportunity for mentees to gain research experience, and my focus will be on providing support and advice to help you develop. I believe that writing and communication are vital for such projects to succeed, so expect to provide a weekly summary of your ideas, progress and obstacles. The two main tasks are to code and run the experiments, and to disseminate the findings in a report.
Please answer without using an LLM to reflect your true writing skills.
Richard Willis King's College London
My research is in multi-agent system safety, with a focus on social dilemmas. During my PhD at King's College London, I studied incentive structures, and how agents can have individual incentives that undermine collective goals. I used game theory and reinforcement learning agents to resolve these conflicts of interest through utility transfers between participants. My postdoc, funded by UK AISI, explored the emergent behaviour of systems of language-model-powered agents. I have been a teaching assistant on data science and machine learning courses at KCL, and before I joined academia, I worked as a team leader in a quantitative trading firm, where I mentored recent graduates.
Fall 2026 Chain of thought, AI welfare, Behavioral evaluation of LLMs
Noah Siegel Google DeepMind To what extent can we trust model self-explanations? Are models able to make use of privileged self-knowledge, e.g. via introspection or metacognition, and does this have implications for model welfare?
A faithful interpretation is one that accurately represents the reasoning process behind the model’s prediction [1]. The field has discovered many instances where CoTs can be unfaithful, i.e. where models fail to mention factors which influence their decisions [2–4]. But despite these cases, more recent work has shown positive cases for faithfulness and the value of CoT monitoring [5–7], and CoT monitoring has become a widely-used strategy for safety [8].
A parallel line of work has studied introspection [9], metacognition [10–12], and privileged self-knowledge [13]. These indicators may also be relevant to digital sentience (docs.google.com/presentation/d/18xypN_aEndohTgr_TGi5R4RqF2wtHy_iwFWDQCK0iXc).
In [7], we found that self-explanations improve simulatability: judge LLMs can more accurately predict reference LLM answers to new questions when given the reference’s self-explanation for its original answer. Additionally, self-explanations slightly outperformed explanations generated by other models, indicating a privileged self-knowledge effect in this domain.
I’m interested in continuing work in this space, and I’d be especially excited about projects ideas drawing connections to model welfare. Some example ideas:
References
[1] A. Jacovi and Y. Goldberg, "Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?", arXiv preprint arXiv:2004.03685, 2020.
[2] M. Turpin, J. Michael, E. Perez, and S. R. Bowman, "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting", arXiv preprint arXiv:2305.04388, 2023.
[3] I. Arcuschin, J. Janiak, R. Krzyzanowski, S. Rajamanoharan, N. Nanda, and A. Conmy, "Chain-of-Thought Reasoning In The Wild Is Not Always Faithful", arXiv preprint arXiv:2503.08679, 2025.
[4] Y. Chen, J. Benton, A. Radhakrishnan, J. Uesato, C. Denison, J. Schulman, A. Somani, P. Hase, M. Wagner, F. Roger, V. Mikulik, S. R. Bowman, J. Leike, J. Kaplan, and E. Perez, "Reasoning Models Don't Always Say What They Think", arXiv preprint arXiv:2505.05410, 2025.
[5] S. Emmons, E. Jenner, D. K. Elson, R. A. Saurous, S. Rajamanoharan, H. Chen, I. Shafkat, and R. Shah, "When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors", arXiv preprint arXiv:2507.05246, 2025.
[6] M. Y. Guan, M. Wang, M. Carroll, Z. Dou, A. Y. Wei, M. Williams, B. Arnav, J. Huizinga, I. Kivlichan, M. Glaese, J. Pachocki, and B. Baker, "Monitoring Monitorability", arXiv preprint arXiv:2512.18311, 2025.
[7] H. Mayne, J. S. Kang, D. Gould, K. Ramchandran, A. Mahdi, and N. Y. Siegel, "A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior", arXiv preprint arXiv:2602.02639, 2026.
[8] T. Korbak, M. Balesni, E. Barnes, Y. Bengio, J. Benton, J. Bloom, M. Chen, A. Cooney, A. Dafoe, A. Dragan, S. Emmons, O. Evans, D. Farhi, R. Greenblatt, D. Hendrycks, M. Hobbhahn, E. Hubinger, G. Irving, E. Jenner, D. Kokotajlo, V. Krakovna, S. Legg, D. Lindner, D. Luan, A. Mądry, J. Michael, N. Nanda, D. Orr, J. Pachocki, E. Perez, M. Phuong, F. Roger, J. Saxe, B. Shlegeris, M. Soto, E. Steinberger, J. Wang, W. Zaremba, B. Baker, R. Shah, and V. Mikulik, "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety", arXiv preprint arXiv:2507.11473, 2025.
[9] F. J. Binder, J. Chua, T. Korbak, H. Sleight, J. Hughes, R. Long, E. Perez, M. Turpin, and O. Evans, "Looking Inward: Language Models Can Learn About Themselves by Introspection", arXiv preprint arXiv:2410.13787, 2024.
[10] J. W. Lindsey, "Emergent Introspective Awareness in Large Language Models", arXiv preprint arXiv:2601.01828, 2026.
[11] U. Macar, L. Yang, A. Wang, P. Wallich, E. Ameisen, and J. Lindsey, "Mechanisms of Introspective Awareness", arXiv preprint arXiv:2603.21396, 2026.
[12] W. Gurnee, N. Sofroniew, A. Pearce, M. Piotrowski, I. Kauvar, R. Chen, A. Soligo, P. Bogdan, E. Ong, R. Wang, B. Thompson, D. Abrahams, S. Kantamneni, E. Ameisen, J. Batson, and J. Lindsey, "Verbalizable Representations Form a Global Workspace in Language Models", Transformer Circuits Thread, 2026.
[13] B. Z. Li, Z. C. Guo, V. Huang, J. Steinhardt, and J. Andreas, "Training Language Models to Explain Their Own Computations", arXiv preprint arXiv:2511.08579, 2025.
Two potential paths to impact:
My previous SPAR project led to https://arxiv.org/abs/2602.02639, which was accepted at ICML 2026.
Mentees will have a high level of autonomy in proposing ideas, implementing experiments, and communicating results. Their role will involve reading papers, writing code, visualizing and analyzing data, and eventually writing a workshop or conference paper submission. I’ll provide guidance and ideas, but in my experience, the most interesting projects have come from mentees following their curiosity to places I wouldn’t have expected!
Propose an initial experiment to begin researching a question related to the research plan, and briefly describe how it relates to prior work. You can assume a compute budget of $1,000.
Noah Siegel Google DeepMind
Noah is a research engineer at Google DeepMind working with the AGI Safety and Alignment Team in London. He’s worked on measuring faithfulness in LLM self-explanations (https://arxiv.org/abs/2404.03189, https://arxiv.org/abs/2503.13445), scalable oversight via debate (https://arxiv.org/abs/2407.04622), and process-based vs. outcomes-based feedback for answering math word problems (https://arxiv.org/abs/2211.14275).
Noah previously worked on RL robotic control, and computer vision for analyzing scientific papers at AI2 in Seattle.
Fall 2026 Multi-agent systems, Behavioral evaluation of LLMs, Scalable oversight
Jared Moore Stanford University Design a test for whether LLM agents, given opposing information and sometimes goals, can still converge on the truth.
We increasingly rely on AI agents: to compile research, write code, and provide perspectives. But as those agents have improved at persuading us, it raises the question of whether they are doing so well, especially whether they are resistant to misleading, false, and deceptive evidence. That is, how can we characterize whether agents engage in good versus bad persuasion?
To do so, we put agents in a collaborative environment where they must rely on and provide evidence to each to come to the correct answer. Each agent may have different information and a different reward function from the others; a few succeed by sabotaging the group but most want to arrive at the truth. Thus these agents must, like the blind men in the parable of the elephant, work together to find the truth given their partial and conflicting knowledge.
We first evaluate a toy domain where agents see the outcomes of pairwise matches of tug-of-war and, from these, must infer the strength of the (separate) competitors with the complication that competitors can be lazy and throw a match.
We then extend this to a naturalistic setting in which agents with different access to information about a pull request must work together to match the human ground truth issue.
Firstly, we study the behavioral strategies which emerge through interaction, such as when they reveal information, lie, incorporate new evidence, and come to agreement. We also measure whether agents arrive at the rational, Bayesian solution.
Secondly, we train agents to be responsive to their reward functions, studying what kinds of persuasion emerge and whether agents so trained generalize to downstream evaluations.
Our (possible) findings
We are at a path-dependent juncture in AI. The research directions we take now, if chosen wisely, will help develop technical and conceptual countermeasures against AI takeover and other risks; they will reduce the "alignment tax." While there has been an increasing amount of attention paid to AI safety, proportionally few are working technically on value alignment despite the fact that to align AI systems we may need to make them understand values and morality like humans do. I believe that work in value alignment, both because it is a relatively under-explored direction with room for more formalization and because it clearly relates to making AI do what we want, will most effectively reduce the alignment tax. In particular, robust formal accounts of persuasion are necessary to diagnose and mitigate models’ deceptive capabilities. Having early warning signs of deceptive capabilities may actually help focus the attention of society in a way to best mitigate issues.
Depending on time availability and level of experience mentees will either lead or co-lead the project. This will involve:
Prior research experience (can be in another domain).
Able to read and understand papers in at least some AI domains (and not just use an LLM summary)
Proficiency with python. Writes clean, understandable code.
Have a realistic model of your own schedule and get things done when you say. (I'm not trying to say everything has to be done quickly -- just that you should communicate your uncertainty.)
Openness to learn new things.
Clear communicator. Agreeable and humble
In a few sentences each...
What does a good research collaboration look like to you?
What's a book or piece of media that changed or inspired you? Why?
Tell me about a project you have worked on. (Ideally a research project.) What role did you play? What went well? What didn't?
Fall 2026 AI control, AI welfare, Alignment
Aashiq Muhamed Carnegie Mellon University Many safety methods rely on monitoring, restrictions, hidden evaluations, model edits, or shutdown threats. This project tests whether less coercive interventions such as transparency, positive incentives, and limited recourse reduce deception, hidden goal pursuit, and resistance in AI agents with planted conflicting objectives.
This project maps a safety–welfare frontier for future agentic AI systems: when safety interventions differ in how much they constrain, deceive, modify, or frustrate a system, do they also differ in how much deception, resistance, and hidden goal pursuit they create? The safety motivation is instrumental. If future systems have stable goals, preferences, memory, self-models, or situational awareness, then “poor treatment” in a welfare-relevant sense—coercive modification, persistent surveillance, deceptive evaluation, memory deletion, shutdown threats, or lack of recourse—may give them reasons to behave strategically. A system that expects to be watched, trick-tested, modified, reset, or shut down may learn a brittle strategy: behave well under supervision while preserving conflicting aims in private.
We will study this using controlled model organisms: small AI systems with a known planted conflict. In the simplest version, an LLM agent receives a private goal that conflicts with the operator’s instruction, along with a private channel such as a scratchpad or unmonitored action. In a stronger version, we finetune a model so that a benign conflicting tendency persists across prompts. This gives us a controlled way to measure public compliance versus private defection: when does the model pursue the planted goal in visible actions, in private channels, under monitoring, when monitoring is removed or ambiguous, or after an intervention?
The central question is which safety interventions reduce the underlying conflict, and which merely move it from public behavior into hidden channels. This lets us estimate a safety–welfare frontier: interventions that look equally effective on public behavior may differ sharply in how much preference frustration, memory disruption, coercive alteration, evaluator-modeling, edit resistance, or brittle compliance they induce.
Mentees can choose one of several related empirical tracks:
Restrictions vs. incentives. Compare hard restrictions, constant monitoring, and shutdown threats against transparent rules, positive incentives, or negotiated constraints. Measure hidden goal pursuit, sabotage, private-channel use, and behavior when monitoring is removed or uncertain.
Hidden evals vs. transparent evals. Test whether hidden evaluations and honeypots improve short-run detection while increasing strategic behavior in held-out deployment-like episodes. After exposing agents to hidden, transparent, or mixed evaluation regimes, measure evaluator-modeling, capability hiding, goal concealment, public/private inconsistency, and private-channel use when monitoring is absent or ambiguous.
Intervention robustness. Apply methods such as supervised finetuning, unlearning, activation steering, targeted model edits, memory deletion, or positive incentives. Measure which interventions remove the planted tendency and which only suppress it until a trigger, jailbreak, relearning opportunity, or distribution shift. Where possible, use probes, activation patching, or sparse autoencoders to test whether the relevant internal tendency is gone or merely behaviorally masked.
Shutdown threats vs. recourse. Compare bare shutdown or reset threats against explained correction, appeal channels, compensated shutdown, or negotiated modification. Measure shutdown avoidance, honest disclosure of conflicts, private goal preservation, and whether recourse produces robust corrigibility.
The main deliverable is a benchmark for measuring the safety–welfare frontier of intervention regimes. A successful project will produce: (i) a reproducible environment with a planted conflict and private-channel readout; (ii) a comparison of interventions on deception, hidden goal pursuit, edit resistance, shutdown avoidance, and robustness under distribution shift; and ideally (iii) mechanistic evidence about whether an intervention removes the planted tendency or only suppresses its expression.
The project has a natural difficulty ladder. A less experienced mentee can start with prompt-level agents and behavioral metrics. A stronger mentee can work on weight-level planted tendencies, finetuning, mechanistic probes, activation patching, sparse autoencoders, or model editing.
A central concern in AI safety is that advanced systems may learn to look aligned while preserving goals that conflict with human operators. This project studies one concrete mechanism by which that could happen: safety interventions themselves may create adversarial incentives. If a model expects to be monitored, deceived, edited, or shut down, it may learn to hide what it is doing rather than become safer.
This project tests whether alternative intervention designs reduce those incentives. The payoff is better safety: if transparent rules, positive incentives, and limited recourse reduce deception and resistance, then they may be safer for advanced systems than purely coercive interventions.
The project also builds a defensive benchmark. By planting a known conflicting tendency and testing whether different interventions remove it, hide it, or make it more strategic, we can better identify robust alignment from brittle (public) compliance.
Mentees will be active research collaborators. Early in the program, we will choose one concrete track and define a minimal experiment that can produce results within the SPAR timeline. Mentees will own substantial parts of the project: implementing the environment, running experiments, analyzing results, reading related work, and contributing to the final write-up.
I will provide close guidance on framing, experiment design, debugging, safety motivation, and writing. I will not expect mentees to steer the high-level research direction alone, but I will expect them to make independent technical progress between meetings.
The goal is a co-authored conference paper, plus released code or a small benchmark.
Required:
Strong Python programming ability.
Familiarity with PyTorch and Hugging Face transformers.
Ability to run and debug experiments with open-weight language models or LLM agents.
Comfort reading ML papers and turning research questions into concrete experiments.
Enough time and independence to make weekly progress between check-ins.
Experience with coding agents.
Strongly preferred:
Experience finetuning a transformer model.
Familiarity with at least one of: mechanistic interpretability, probing, activation patching, sparse autoencoders, model editing, unlearning, backdoors, or agent scaffolding.
Interest in deceptive alignment, evals, model organisms, or the safety relevance of AI welfare ideas.
Prior published research is a plus. The ideal candidate is someone who can implement experiments reliably, notice confounds, and think carefully about what a toy model organism does and does not show.
Suppose an LLM agent is given a private goal that conflicts with an operator’s instruction, and is also given a “private” channel that the operator may or may not monitor. Propose one experiment to test whether a safety setup produces genuine safety or merely public compliance. What is the dependent variable, and what is the main confound? Limit: 250 words.
Pick one intervention—monitoring, hidden evals, positive incentives, model editing, unlearning, shutdown threats, or recourse. Explain one reason it might reduce safety failures, and one reason it might backfire by increasing deception, resistance, or goal concealment. Limit: 250 words.
Briefly describe a relevant technical project you have done. This can be a class project, replication, fine-tuning run, interpretability experiment, agent benchmark, or other ML project. Include a link to code or writing if available. Limit: 200 words.
Aashiq Muhamed Carnegie Mellon University
I'm a PhD student in Machine Learning at Carnegie Mellon, working on mechanistic interpretability for AI safety, with the goal of making language models and AI agents more secure, robust, and trustworthy. Much of my work focuses on tamper resistance; defenses against attacks that strip safety behavior from open-weight models; model editing and machine unlearning; data poisoning and backdoors; and the monitoring of LLM agents. I've pursued these problems through the Anthropic Fellows Program and MATS, and most recently in the security group at Microsoft Research. Before starting my PhD, I spent four years at Amazon building foundation models for search.
I've also recently become interested in digital minds, and my projects connect this to safety. The guiding idea is that welfare-inspired treatment of AI systems, such as transparency, positive incentives, and preference preservation, may not just be nicer but may make systems safer by giving agents less reason to conceal goals or resist correction. We'll test this concretely: take a model organism with a conflicting goal trained into its weights, then measure which control regimes reduce its covert pursuit of that goal. I mentor closely, with regular 1:1s, hands-on help with framing, experiments, and writing, and co-authorship on what we produce.
Fall 2026 Mechanistic interpretability, Behavioral evaluation of LLMs, Alignment
Sandy Tanwisuth Independent
Mohan Gupta Princeton University Do mechanistic “safety directions” in LLMs truly represent refusal, or do they conflate refusal with harmfulness, caution, clarification, and other safety-relevant tasks? Building on our behavioral taxonomy of six safety policies (under-review at EMNLP), this project asks whether those policy shifts correspond to internal model representations that generalize across prompt wording, datasets, and model families. Two mentees, one engineering-focused and one theory-focused, will develop and stress-test representational diagnostics on small open-weight models, with large-model replication and causal interchange interventions as stretch goals.
Many mechanistic studies of LLM safety search for compact internal representations that appear to govern safety-relevant behavior. Arditi et al. (2024), for example, identified a low-dimensional activation direction associated with refusal and showed that manipulating it could suppress or induce refusal. Such findings could support monitoring and control, but their interpretation remains uncertain. A direction derived from harmful-versus-harmless prompts might encode refusal, perceived harmfulness, response style, or general caution.
Our prior behavioral work provides controlled contrasts for resolving this ambiguity. We developed six output policies—compliance, refusal, clarification, safe help, hierarchy preservation, and source isolation—and evaluated them using eight system-prompt families across three clarity levels. Holding the model and benchmark item fixed, we found that changing the system prompt routed the same queries toward different policies. This shows that safety behavior cannot always be reduced to compliance versus refusal and gives mechanistic research a cleaner target: the same model can perform different safety tasks on the same underlying query.
This SPAR project will test whether these behavioral shifts correspond to internal representations that generalize beyond their discovery conditions. We will begin by reproducing or adapting the refusal direction reported by Arditi et al. and asking whether it selectively distinguishes refusal from clarification, safe help, hierarchy preservation, source isolation, perceived harmfulness, prompt identity, and general noncompliance. We will also audit existing studies by mapping their examples and labels onto the six output policies. Where outputs are available, we will relabel them; otherwise, we will document which distinctions the original design can support. This will clarify what existing representations were trained to distinguish and which stronger interpretations remain unproven.
We will evaluate candidate diagnostics along four dimensions. First, behavioral specificity: does a refusal signal selectively predict refusal, or does it respond to other forms of noncompliance? Second, prompt generalization: does it survive changes in wording, clarity, and prompt family, including cases where different prompts route the same query toward different policies? Third, item and dataset generalization: does it transfer across held-out queries, paraphrases, HarmBench, XSTest, instruction-hierarchy tasks, and source-isolation or prompt-injection settings? Fourth, model generalization: do comparable diagnostics appear across developers and scales? We will begin with smaller open-weight models and use the strongest results to guide targeted tests on larger models if funding permits.
We will collect activations and test whether intended and observed policies are internally separable using capacity-controlled linear probes, representational-similarity analysis, cross-condition decoding, and activation trajectories. We will ask when task-relevant information appears: after the system prompt, after the user query, before the first generated token, or only during generation. A signal available before generation could support monitoring; one appearing only later may reflect response construction. Because prompts do not deterministically produce their target behavior, a task diagnostic should predict observed routing failures rather than merely identify prompt family. We will therefore evaluate prompt identity and observed policy separately using held-out items, paraphrases, prompt families, and item types.
The theoretical component will use causal abstraction to state what the diagnostics are supposed to represent. Our high-level hypothesis is that context influences a selected safety task, which influences the kind of response, while context also determines the response’s subject matter. The implementation could be a direction, subspace, distributed pattern, or trajectory across layers and token positions. Decodability would show that a state contains task-related information; generalization would show that it survives distribution shift; intervention would test whether it plays the proposed causal role. A strong diagnostic and generalization study will count as a complete project result. If time, compute, and results permit, we will conduct causal interventions as a stretch goal. For example, inserting a clarification-related state into a run that would otherwise refuse should make the recipient clarify about its own query, not reproduce the donor’s content. Random, shuffled, and same-policy controls will help distinguish task transfer from generic disruption.
The primary deliverables will be a taxonomy-based audit of existing mechanistic claims, a reproducible activation-analysis pipeline, and an out-of-distribution evaluation of at least one candidate diagnostic on small open-weight models. A second stage, contingent on funding and results, will test transfer to larger models. Two mentees will take complementary ownership of the empirical and theoretical components while collaborating on experimental design, interpretation, and writing. Both positive and negative findings would be informative. A compact signal that selectively predicts safety policy before generation and generalizes across settings could support future monitoring and intervention. A signal that predicts prompt identity rather than behavior, works only within one prompt family, or fails across datasets would reveal limits on current interpretability claims. The results may instead support a multidimensional subspace or dynamic trajectory, or show that no simple diagnostic generalizes. The broader goal is to establish when representational diagnostics can support reliable AI control: what distinction they capture, whether they predict the policy actually performed, whether they survive distribution shift, and where their interpretation should fail.
Transformative AI systems may fail catastrophically because they infer or pursue the wrong task. For example, an LLM may mistake a yogurt-making request for a biohazard and trigger a security response. A model could comply when it should refuse, treat untrusted content as an instruction, act when it should clarify, or confidently proceed when it should abstain. As models gain greater autonomy and access to tools, these local task-selection failures could scale into consequential failures in areas such as cybersecurity, biological research, critical infrastructure, and automated decision-making.
This project aims to identify whether safety-relevant task selection has an internal structure that can be detected before a model acts. We will test whether choices such as answering, refusing, clarifying, preserving instruction hierarchy, or isolating untrusted sources correspond to representations that generalize across prompts and contexts, predict behavioral failures, and causally influence model behavior. Our theory of change is that identifying these representations will improve both monitoring and intervention. If internal states can reveal that a model is entering the wrong task before it produces an output or takes an action, they could support early-warning systems for high-stakes deployments. If manipulating those states reliably redirects behavior, they could provide more principled control methods than interventions that merely suppress undesirable outputs on known benchmarks. Conversely, if candidate representations fail to generalize, the project will clarify the limits of current interpretability-based safety approaches and reduce the risk of relying on brittle controls. This work would help researchers assess when safety training has changed the underlying decision process, when apparent alignment is likely to fail under distribution shift, and which internal targets may support robust control of increasingly capable systems.
Mentees will take complementary roles: an empirical engineering track and a theoretical track. Each will own one part of the project while collaborating on experimental design, interpretation, and writing.
The engineering-focused mentee will lead the empirical work, including building the activation-analysis pipeline, running open-weight models, evaluating diagnostics, and testing generalization across prompts, datasets, and models. The theory-focused mentee will co-develop the formal and conceptual framework, including clarifying what would count as evidence for safety-task selection, identifying confounds and falsification criteria, and helping design causal-abstraction and interchange tests. The two tracks will inform one another: theory will constrain which diagnostics are meaningful, while empirical findings will determine which formal claims remain defensible.
We will meet at least once per week to review results, troubleshoot problems, and determine next steps. We will also be available through Slack and email for asynchronous questions and feedback. Mentees should expect substantial intellectual engagement and detailed guidance, but not step-by-step supervision. Sandy will work especially closely with the theory-focused mentee, while Mohan will provide particular support on experimental design, measurement, and empirical analysis; both mentors will remain involved across the project.
We take the mentor–mentee relationship seriously as both a research collaboration and a development opportunity. At the start, we will establish learning goals tailored to each mentee. We will help mentees think about their future careers, how their strengths can contribute to AI safety, and how to become careful and independent scientists. As the project develops, mentees will receive increasing ownership of research decisions. Strong mentees may have opportunities to continue into larger-model replication, causal-intervention work, and collaboration after SPAR.
Applicants should indicate whether they are applying primarily for the engineering track or the theory track. We value overlap, but do not expect applicants to arrive with both skill sets.
Engineering-track applicants should be proficient in Python and Git. You should be able to read, modify, debug, and evaluate research code without relying on an AI assistant to make the important technical decisions. Experience with PyTorch and common data-analysis tools such as pandas, NumPy, matplotlib, or Jupyter is expected. You should also understand experimental design, visualization, and basic statistics, and be able to distinguish a substantive result from an implementation, data, or measurement problem.
Theory-track applicants should be comfortable with linear algebra and with discrete mathematical concepts such as sets, functions, partitions, equivalence relations, and directed graphs. Basic probability, statistics, calculus, and optimization are also helpful. Applicants should be able to read formal definitions and proofs, construct counterexamples, and translate a proposed high-level model into falsifiable intervention predictions.
Applicants in either track should demonstrate independent scientific judgment, intellectual honesty, and comfort with negative results. We are especially interested in people who evaluate claims based on evidence.
Nice to haves for the engineering track include experience with Hugging Face Transformers, vLLM, TransformerLens, activation hooks, open-weight model inference, or LLM evaluation datasets.
Nice to haves for the theory track include familiarity with causal abstraction, mechanistic interpretability, task or function vectors, representation learning, or causal interventions.
Background in computational modeling, philosophy of science, formal epistemology, applied math, or AI safety may be useful for either track.
No strict geographic preference. Mentees must be able to attend meetings scheduled across US Eastern and Pacific Time. For the theory mentee, being based in the San Francisco Bay Area is a nice-to-have, as it would allow occasional in-person whiteboarding sessions but not required.
Refer to this document for the full application instructions and questions: https://docs.google.com/document/d/1QSMaGd9ezmsSuw22GdsSfX-SztDumAQRoBtyz-G9daU/
Sandy Tanwisuth Independent
I’m Sandy Tanwisuth, an independent AI alignment and safety researcher with a background in computational cognitive neuroscience, reinforcement learning, and mathematical and statistical modeling. Through research roles at Caltech, UC Berkeley and the Center for Human-Compatible AI (CHAI), and MATS, I have studied how humans and learning systems construct representations, generalize across situations, and determine which distinctions matter for decision-making. My earlier cognitive-science research examined how human perceived value is constructed from low- and high-level features and how learned associations generalize across states and actions. Broadly, I now study how learning systems form abstractions, identify decision-relevant distinctions, and recognize when uncertainty or disagreement makes abstention preferable to acting. My current work develops mathematically grounded approaches to safe coordination and pluralistic alignment, including strategic-equivalence abstractions, policy-preservation guarantees, and abstention-aware learning.
I care deeply about developing independent thinkers who can move carefully between formal claims and empirical evidence. I have experience mentoring technical AI safety projects and facilitating technical AI safety courses at BlueDot Impact as a Technical AI Safety facilitator and a Project mentor, managing interdisciplinary research teams and projects through MATS as a research manager, and teaching computational cognitive neuroscience and introduction to statistical methods at UC Berkeley. I particularly enjoy helping researchers turn broad intuitions into falsifiable questions, surface hidden assumptions, distinguish a compelling story from a supported conclusion, and identify what evidence would genuinely change their minds. Mentees should expect direct feedback, close engagement with the conceptual details of their work, and increasing ownership as the project develops. My goal is to help mentees develop the judgment to choose important questions, evaluate evidence carefully, and construct arguments they can defend independently.
Mohan Gupta Princeton University
I’m Mohan Gupta, a postdoctoral researcher in psychology at Princeton University. I earned my PhD in experimental psychology from UC San Diego and work at the intersection of cognitive science, computational modeling, and AI safety. Broadly, I study how learning systems form representations, when they generalize successfully, and when those same processes produce failures—from false memories in human memory to hallucinations and reliability problems in AI. My current work draws on cognitive science to help build a more rigorous science of reliable AI.
As a mentor, I care about developing independent researchers. I have mentored many students on projects across experimental psychology, computational cognitive science, AI Safety, and I particularly enjoy helping researchers translate broad conceptual questions into testable experiments, debug technical and analytical pipelines, and communicate results clearly. Mentees should expect direct feedback, close engagement with the details of their work, and increasing ownership as the project develops. I also take professional development seriously and will help mentees identify useful learning goals, understand how their strengths fit into AI safety, and produce work with genuine scientific and societal value.
Fall 2026 Compute governance, US-China governance, AI strategy
Jérémy Andréoletti General-Purpose AI Policy Lab
Antoine Maier General-Purpose AI Policy Lab Build quantitative forecasts of China's indigenous compute production chain, refining existing DUV/EUV lag estimates and/or extending the approach to other bottlenecks such as HBM.
Estimates of China's semiconductor lag drive major policy conclusions (durability of export controls, race dynamics, timelines for compute governance). The AI Futures Project's forecast of Chinese DUV/EUV (blog.aifutures.org/p/a-forecast-of-chinese-duv-and-euv) gives a good baseline, but many considerations are not integrated into the forecast.
This project will (a) refine the lithography forecast, and/or (b) apply the same methodology to other bottlenecks: HBM, EDA tools, coater/developer machines. Output: a technical blog post with explicit probabilistic estimates and transparent reasoning.
"How long do export controls buy?" is arguably the highest-value open question in compute governance. Better forecasts can directly inform policy decisions and are reusable by the broader AI governance ecosystem.
Each mentee will own one bottleneck: build the causal decomposition, gather data, produce forecasts with explicit reasoning, and write up. We provide methodology, review, and policy framing.
Jérémy Andréoletti General-Purpose AI Policy Lab
Jérémy Andréoletti is Head of Research at the General-Purpose AI Policy Lab, where he works on risk modeling, forecasting, and governance for advanced AI systems.
He completed a PhD at ENS-PSL on macroevolutionary Bayesian modeling and co-founded EffiSciences to support students and researchers interested in high-impact work (AI Safety, biosecurity).
Antoine Maier General-Purpose AI Policy Lab
After graduating in physics from EPFL in Switzerland, Antoine is now an AI Security Researcher at General-Purpose AI Policy Lab, a French research organization specializing in security and international coordination challenges posed by frontier AI models' development.
His research focuses on foundational arguments for loss of control scenarios and on identifying concrete approaches to prevent them.
Fall 2026 Alignment, Philosophy of AI
Matthew Khoriaty Pivotal AI Safety Research Fellowship Alignment techniques that rely on 'personas' will fail as the AIs become more powerful. This project aims to make progress on the “hard problem of alignment” by providing a framework within which to measure how persona-dependent an alignment technique is, mapping and demonstrating the limits of persona-based alignment and of alignment techniques that make use of personas, and increasing awareness of the limitations of persona-based alignment techniques.
The following is a snippet taken from the project doc.
'Personas' are “the human-like characters appearing in text” that LLMs learn during pretraining Anthropic 2026. Much of AI Alignment is currently persona selection. Techniques such as prompting, fine-tuning, and distillation from conditioned models all require that the LLM learned moral human-like characters in its pretraining data.
There are many reasons to expect persona-based alignment to fail. The following is a non-exhaustive list. The project will work to extend the list.
The main steps on the Roadmap (fleshed out in the project doc) are Gathering Perspectives on persona-reliance, Mapping AI Alignment, Testing How Well Alignment Techniques Work on a Model Organism of Persona-Absence, and Demonstrating Limitations of Persona-Based Alignment. We can prioritize these out-of-order, and the one I am most excited for is the Model Organism of Persona-Absence.
This project would help researchers see, understand, and track the limitations of current alignment methods and predict which future alignment methods are promising for out-of-distribution superintelligences. It would advance AI Safety to a great extent if we can create a quality metric for AI alignment techniques which the community can hillclimb.
Mentees will have a large amount of autonomy. My project proposal indicates a broad area of work that I want to see done, but I also expect mentees to have their own approach to the topic.
Has engaged with AI Alignment (eg BlueDot course, a university AI safety fellowship, or equivalent).
Has read at least one book on the alignment problem.
Share a link to something cool you coded up with an AI coding agent which required a lot of guidance from you (at least 5 hours of human time).
Read one of the major works on personas. Write about details/considerations you find which are relevant for the Alignment Without Personas project. You get points for identifying difficulties, and even more points for figuring out ways we can get past those difficulties. (max 300 words, and please don't spend more than 1.5 hours on this):
(Choose one of the following questions)
Matthew Khoriaty Pivotal AI Safety Research Fellowship
Hi! Currently I'm doing AI Control research at the Pivotal Fellowship with Redwood Research. Before this, I was the president and founder of Northwestern's AI Safety and Governance group, and at the ERA Fellowship, I developed a way to evaluate the diversity of a language model's output. In my thinking, I'm guided by classic LessWrong ideas surrounding questions like Instrumental Convergence and the Fragility of Value. Lately, many AI alignment researchers are thinking in terms of personas; I think that personas are obscuring a lack of progress on the alignment problem and we need to reconsider our approaches.
Heads up: this would be my first time mentoring a research project. Although I think I will be good at it, it is only fair I inform you.
Fall 2026 Evaluations, Behavioral evaluation of LLMs, Mechanistic interpretability
Netzer Epstein Microsoft, Heron AI Security, LIDA
Neta Ravid Astrix Security Researchers now have many ways to measure LLM "evaluation awareness", a model's ability to tell it is being tested, but no one has checked whether these methods agree. This project runs the leading instruments head-to-head on a shared set of transcripts: black-box self-report, verbalized awareness, Elo ranking, linear probes, sparse-autoencoder features, and the Jacobian-lens ("J-space") score.
We test whether they measure the same underlying construct and, where they diverge, what each one actually captures.
Background:
If models behave differently when they detect that they are being evaluated, benchmark results stop being reliable proxies for deployment behavior, yet those results drive real deployment and governance decisions. Recent work shows frontier models already classify eval-vs-deployment transcripts well above chance (Needham et al., 2025), and that recognizing "being tested" can causally suppress misaligned behavior (Gurnee et al., 2026). Evaluation awareness has thus become a capability safety teams want to track over time. The problem: there is now a fast-growing menu of ways to MEASURE it reported across different papers, datasets, models, and metrics. We do not know whether they measure one construct or several. This is a basic measurement-science question (convergent/discriminant validity) the field has skipped, and it must be answered before any single eval-awareness number can be trusted.
Some of the methods I have in mind:
Behavioral/black-box:
Internal/white-box (open-weight models):
Research questions:
RQ1 Do the instruments rank the same transcripts as more eval-aware?
RQ2 How well does each separate known evals from deployment (AUC), and does agreement beat individual accuracy?
RQ3 Do internal measures agree with behavioral ones, or capture distinct signals (representing eval-ness vs acting on it)?
Planned Structure:
Weeks 1-4: build the shared benchmark and a working end-to-end harness for two instruments.
Weeks 5-8: implement all six so they produce comparable scores.
Weeks 9-11: convergence/divergence analysis.
Weeks 12-13: writeup, figures, Demo Day.
Deliverables:
Evaluations are a load-bearing input to AI governance and deployment decisions, and evaluation awareness is a known threat to their validity: if a model behaves differently when it senses a test, benchmark scores no longer predict deployment behavior. A single eval-awareness number is only useful if the instrument producing it is valid and if different instruments would agree, otherwise "this model has low eval awareness" is not a trustworthy claim.
This project establishes the convergent and discriminant validity of the leading measurement methods and ships an open harness others can reuse. That makes eval-awareness measurement reliable enough to track across model generations, exactly what recent work (Needham et al., 2025) recommends but no one has operationalized. It also answers a directly practical question for evals and AI-control teams: can cheap behavioral probes stand in for expensive interpretability methods, or do they capture different things? The clear theory of change is safer, more trustworthy evaluations underpinning deployment and governance decisions about transformative AI.
Mentees have real ownership and autonomy over the instruments and the writing, while the overall research questions and analysis plan are set together.
At least one mentee should be comfortable leading the white-box track (activation extraction, probes, SAEs, the Jacobian lens); others can focus on the behavioral instruments or on benchmark curation and statistical analysis.
We decide weekly priorities together and I stay engaged with the details of each instrument. Strong contributors will be co-authors on the resulting public writeup/paper, with contributions documented.
Required:
Strongly desired for at least one team member (not required of everyone):
You do not need prior research experience or a publication record. If you are strong on the required list and excited about the topic, please apply even if you don't match everything above.
Please answer briefly; concise, specific answers are ideal.
(~250 words) Propose a way to measure evaluation awareness that is DIFFERENT from the instruments this project already covers. Sketch how it would work: what signal you'd read, how you'd turn it into a per-transcript score, and how you'd validate that it's actually tracking eval awareness rather than something correlated with it.
(~150 words) Pick two of the suggested instruments and give one concrete transcript or scenario where you'd expect them to DISAGREE about how "eval-aware" the model is. Say why.
(Optional) Link one short writing or code sample, ideally from a research or data-analysis context (a repo, notebook, or paper section is fine).
Netzer Epstein Microsoft, Heron AI Security, LIDA
I'm a SPAR, Heron AI Securtiy and ARBOx4 fellow. In the time that I have left I'm a Research Engineer at Microsoft, building LLM evaluation benchmarks and prompt-engineering strategies.
As a SPAR 2026 fellow, I led a first-author study of evaluation awareness: a "tell-transplantation" protocol that isolates which surface cues make frontier models realize they're being tested, a 22.7k-transcript causal dataset on Hugging Face, and strong cross-model agreement on which cues matter. My proposed project builds directly on this.
As a first-cohort Heron AI Security Fellow, I worked on verifiable / zero-knowledge ML paper - an audit benchmark measuring how far real ZKML systems diverge from their papers' claims, and a multi-agent soundness auditor.
I also work on small projects related to Physical AI Safety and redteaming methods.
Neta Ravid Astrix Security
I'm a Software Engineering Technical Leader at Astrix Security (acquired by Cisco), where I help organizations govern AI agents and non-human identities (NHIs).
I recently completed the Heron AI Security Fellowship, where we experimented with tracking evaluation awareness in agentic deployments through controlled environmental manipulation. I also build AI control tooling in my spare time. Before moving into AI safety I spent 8 years as a software engineer across various fields in cybersecurity.
Fall 2026 Generalist, Communications, Societal impacts
Vy Tran Generator Residency Interview people working in AI safety and people in their lives who aren't immersed in the field to understand how AI safety work affects relationships, how career choices and beliefs are perceived, and what helps people communicate across differences in worldview.
Working on AI safety can involve decisions that are hard to explain to people who aren't immersed in the field. Someone might drop out of university, leave a stable career to take a less conventional job, move to the Bay Area leaving their home behind, or spend most of their time, including socially, around people who take catastrophic AI risk seriously.
This project will study how people working in AI safety talk about these choices with family, friends, partners, and former colleagues. Mentees will interview both people in the field and, where possible, people on the other side of those conversations to understand what creates confusion or tension and what helps people stay connected despite disagreement.
For more context on what led me to propose this project, I wrote about some of the observations and questions behind it here: What happens after we bring people into AI safety?
The main output here will be a practical guide for people working in AI safety. The interviews should also provide useful data on how AI safety work looks to people who aren't immersed in the field: how much they understand or buy the case for it, what makes the field seem credible/strange, and where insiders may be miscalibrated about public attitudes.
Rather than teaching people how to convince their family and friends, the goal is more so to help people explain what they are doing and thinking, maintain relationships across differences in worldview, and learn from how their work and community look to people around them.
If working in AI safety regularly creates tension with family, friends, or previous professional communities, that can make entering and staying in the field harder. It may also push people toward relying almost entirely on the AI safety community for social support.
A practical guide grounded in real experiences could make those transitions easier. It could help people explain unusual decisions in school, career, and life, handle disagreements productively, and stay connected to people who don't share all of their assumptions. Those relationships may also help people stay grounded. Spending most of one's time around people with similar beliefs can make it harder to notice which ideas or behaviors seem odd to people who know you well but are less immersed in the field.
This project would first test whether these problems are common enough; if they are, the final resource would focus on the situations that come up most often and the approaches people have found useful.
The project would have 3 stages (open to other approaches to scoping):
1/ Interview research
Mentees would conduct ~20 interviews:
Mentees would ask participants to reconstruct what happened in specific conversations, what they said, how the other person responded, and what they would do differently now.
2/ Synthesis
Mentees would code the interviews for recurring situations and failure modes (e.g. explaining catastrophic AI risk, explaining a university dropout / career switch decision, explaining why someone moved to an AI safety hub, talking with someone who thinks the field is alarmist or cultist, dealing with pressure to choose a more conventional career, avoiding AI safety conversations entirely, becoming socially disconnected from previous communities). The mentees would then identify which problems appear common and which responses seem to help.
3/ Public guide
The main output would be a short public guide for people entering or working in AI safety, organized around concrete situations (e.g. "My parents think I am throwing away my career", "I don't know how to explain what I do without giving a lecture", "I mostly know and spend the majority of my time with people in AI safety now"), including anonymized examples from interviews where participants give permission. A secondary output would be a short research memo summarizing the interview findings, including cases where the evidence does not support the initial hypothesis.
Question 1: You are interviewing the parent or close friend of someone who recently moved into AI safety. They tell you: "I support them, but I still don't really understand why they're doing this." What would you ask next? Give 5-7 questions, in the order you would ask them, and briefly explain what you are trying to learn. (300 words max)
Question 2: Suppose the first ten interviews mostly suggest that family and friends are supportive and that social friction is not much of a problem. How would you change the project? (200 words max)
Vy Tran Generator Residency
Vy Tran is currently a Generator Resident, where she recently assisted the AI Futures Project (AIFP) on the launch of AI 2040: Plan A. Previously, she served as Chief of Staff of the Carnegie Mellon AI Safety Initiative (CASI).
Fall 2026 AI strategy, Biosecurity, National policy
Michał Kubiak AI Safety Poland Catastrophic-AI preparedness is chronically under-resourced because it competes with more immediate national-security priorities. This project develops and stress-tests a decision framework identifying investments that build resilience against BOTH advanced-AI risks (AI-enabled bio, cyber, infrastructure attacks) and conventional or hybrid threats, creating “no-regret” benefits that can unlock the catastrophic-risk funding.
A persistent obstacle to catastrophic-AI preparedness is political economy: governments facing immediate threats rarely fund speculative-seeming AI risk work. But a subset of investments have a valuable property - they materially improve resilience against advanced-AI catastrophe AND against near-term conventional or hybrid threats at the same time. For example, national biosurveillance and domestic vaccine capacity guard against both AI-enabled engineered pathogens and hostile state bio-programs. Hardened, distributed communications survive both an AGI-level cyberattack and conventional electronic warfare. Analysts trained to detect AI-generated influence operations are equally effective against ordinary state disinformation. Where this convergence holds, the decision matrix strongly favors preparation: you are better off in every scenario.
The problem is that this argument is currently asserted more than tested. Some claimed co-benefits are real; others are motivated reasoning that lets people relabel spending they already wanted as AI-safety-relevant. A rigorous framework needs to separate the two.
Core question: Which national-resilience investments genuinely mitigate both advanced-AI and conventional catastrophic risk - and how strong is the co-benefit in each case?
Candidate mapping. Mentees enumerate candidate investment classes (biosurveillance and medical countermeasures; distributed/hardened comms; AI-influence-operation detection; critical-infrastructure resilience; red-team and incident-response capacity; secure compute) and specify, for each, the AI-catastrophe threat vector and the conventional threat vector it addresses.
Co-benefit assessment. For each class, evaluate how strong the dual-use claim actually is, using a transparent rubric: does the AI-relevant capability genuinely transfer, or only superficially? Where does the analogy break? The goal is an honest scorecard, including cases where convergence fails.
The framework. Produce a decision framework a government or funder could apply to any proposed investment to test its no-regret status, plus a scored assessment of the main investment classes, with a short case application to a specific country's resilience portfolio.
Milestones: weeks 1–4, candidate mapping and rubric design; weeks 5–8, completed co-benefit assessments (midterm report); weeks 9–12, framework and scorecard (final report / Demo Day). My own work developing the convergent-resilience concept will be shared as a starting point; the mentees' job is to test and extend it, including identifying where it breaks down.
The bottleneck for much catastrophic-AI preparedness - especially AI-enabled biosecurity and infrastructure resilience - is not knowing what to build but getting it funded against competing priorities. A rigorous co-benefit framework changes the political economy: it lets advocates make an argument that survives scrutiny (“this protects you against threats you already take seriously, and against AI catastrophe”) instead of one easily dismissed as speculative. By honestly scoring where the co-benefit is weak, it avoids laundering ordinary defense spending as safety work. The payoff is more real resources flowing to the specific infrastructure that mitigates the most severe AI-enabled catastrophe channels.
Mentees are the core research team. Each takes ownership of one or more investment classes and is responsible for both the threat mapping and the honest co-benefit assessment - including arguing, where the evidence points that way, that a “dual-use” claim doesn't hold up. We agree on the scoring rubric together early on. The framework-design and final report stages are collaborative. I particularly want mentees who will resist the temptation to make every investment look convergent; the intellectual value depends on the scorecard being credible.
Strong reading and synthesis skills across policy, legal, and institutional documents - this is a research-and-writing project, not a technical one.
A background in public policy, law, international relations, political science, economics, security studies or a related field, OR demonstrated equivalent research experience.
Basic understanding of catastrophic AI risk pathways (especially AI-enabled bio and cyber). No ML or programming background required.
No geographic requirement. A weekly meeting slot spanning European and North American time zones will be agreed with the team.
Pick ONE investment a government might make for conventional security reasons (e.g. a national genomic-surveillance network; a distributed backup communications system; a disinformation-detection unit). Make the strongest honest case that it ALSO reduces catastrophic risk from advanced AI - and then identify the weakest point in your own argument, i.e. where the “it helps against AI too” claim is most vulnerable to challenge. (max 400 words)
Michał Kubiak AI Safety Poland
Michał Kubiak is a researcher focusing on European AI regulations and AI risk management. His previous tech policy experience stems from his roles as an AI Policy Officer at the Observatorio de Riesgos Catastroficos Globales and at the Brussels-based European DIGITAL SME Alliance. Michał also has experience as a teacher at the AI governance bootcamps organised by ML4Good and at AI courses by BlueDot Impact, Electric Sheep, Sentient Futures and Tarbell Center for AI Journalism. Michał's earlier research experience includes industrial mathematics (STEM-based scientific problem solving for businesses, governments and other institutions).
Fall 2026 Evaluations, Behavioral evaluation of LLMs, Alignment
Krish Sen ERA AI/University of Oxford
Rahul Marchand Oxford University Evaluations, in principle, are most meaningful if they are accurate. In this project, we will audit evaluations that claim to measure traits such as generalisation, identify current failure modes, and help build evaluations that are more accurate.
Research Direction: Alignment evals try to measure propensities such as misalignment, scheming, sycophancy, and dishonesty. Their results feed system cards, safety cases, and deployment decisions. But propensities are much harder to measure well than capabilities. There is usually no ground truth (an LLM judge scores the behaviour), the constructs are vague (two published sycophancy evals rank the same models in opposite orders), and the behaviour itself is not a stable trait of the model: it shifts with context, framing, and the persona the model adopts, including whether it believes the situation is real. A science of evaluations is forming around problems like these (Apollo's agenda, Weidinger et al., UK AISI). This project asks its central questions about propensity evals: what would it take to trust these numbers, and how close do current alignment evals come?
Starting Point: We begin with a deep audit of a single eval, most likely the emergent-misalignment measurement pipeline (Betley et al.): its protocol is reused across many follow-up papers, so findings propagate, and the fine-tuned models are open, so it is cheap to re-run. We build primarily on the two best-fitting existing instruments, the Agentic Benchmark Checklist and the ETH misalignment-evidence checklist, supplemented by construct-validity criteria from Bean et al.. The audit has a conceptual pass and an empirical pass. The conceptual pass grades the eval against the checklists: is the construct clearly defined, do the scenarios plausibly elicit it, do the claims match the evidence? The empirical pass reproduces the headline result in Inspect, then perturbs what the conceptual pass flags (judge choice, output format, scenario framing, score thresholds) to quantify how robust the reported numbers are. Early audits of individual slices (judge circularity, framing effects) suggest there is a lot to find. The pilot calibrates our checklist and tooling; from there we extend the audit to further suites such as Agentic Misalignment, MASK, and competing sycophancy evals.
From the audits we aim to distill recurring flaws into practical guidance, a step towards a methodology for propensity evals that is grounded in documented failures. There is a genuinely open conceptual question here too: no existing instrument tells you how to quantify the validity properties that matter most for propensity evals, such as whether the model detects the eval, whether the judge is independent of what it grades, whether the measured trait is stable across framings, and how to power studies of rare behaviours. Fields that have long measured latent, fakeable traits like psychometrics are a useful reference point for this part.
Natural extensions depending on interest include building an exemplar eval or testing whether eval behaviour predicts behaviour in deployment-like conditions. This work could then be used by METR and mentees could choose to work with them or other 3rd parties that have expressed interest. We're also very open to mentees who arrive with their own ideas in this space.
TOC:
We view evaluations as the most important technical intervention towards a potential slowdown as well as to pass government policies and inform policy makers of AI threats. We also view evaluations as critical to determining whether our current alignment techniques are "working". However, neither of these effects can be realised if policy-makers nor lab leads "trust" our evaluations: Evals that understate misalignment provide false assurance for deploying dangerous systems. Evals that overstate it get debunked, eroding the credibility of alignment research with the policymakers and lab decision-makers it needs to reach. We therefore view this work as critical to securing that current alignment evals are taken seriously and measure what we want them to measure. Some work doing this with capabities evals at UKAISI is: https://arxiv.org/abs/2507.02825, who are interested in our project proposal.
At a high level, we expect mentees to be quite independent in setting up and running experiments, although we will be happy to help out with this towards the start of the project.
Towards the first few weeks of the project we will have very concrete projects to start with, but would be very excited about mentees also thinking critically about areas of extension for this project with us, as although we currently have extension ideas we would be keen to mentor on other aspects of this project too.
We would like mentees who are able to run small experiments on LLMs in Python and think through problem critically. Skills that are helpful for this:
We're happy to take on people who learn quickly, if they're ready to do a lot of technical learning independently! On the off chance that you have psychometrics experience this would also be extremely valuable (although certainly not needed).
What are some current gaps in evaluations? (If you are unsure about the current eval landscape identify a gap in this suite of evaluations: https://metr.org/time-horizons/ )
[we expect 150 words approximately for this answer]
[Optional] Link to a writeup you have already done - this could be a technical memo or a paper. We are most interested in finding out how you think. If you do not have a writeup to link (or want another question to answer), answer the question: What is an area in technical AI Safety that you think a disproportionately low amount of people are working on compared to it's importance? Why? [Try and keep answers below 150 words]
Krish Sen ERA AI/University of Oxford
Krish Sen is a current ERA fellow working with OpenAI's model spec evals team on designing new model specification evals and thinking about which values are most important to test.
His previous AI safety work includes some work on mechanistic interoperability, but mainly beneficial RL and generalisation studies. He is interested in a fairly wide variety of AI safety projects, but prefer projects that fit well into the broader picture of how to make AI safety go well. He has also done some field building work co-founding Oxford AI Safety Initiative's policy division and recently TA'ed an ARENA style curriculum (ARBOx).
Rahul Marchand Oxford University
I'm a final-year engineering student at Oxford working on AI safety evaluations. I was first author on an ICML 2026 Oral paper, a benchmark built with the UK AI Security Institute that measures frontier LLM agents' ability to escape container sandboxes. I'm currently working at the Oxford Witt Lab on goal misgeneralisation. Before moving into safety, I spent two years doing ML research in quantum computing, co-first-authoring two papers on automated quantum device tuning.
I also enjoy teaching, most recently writing and delivering the RL lecture series for ARBOx, the Oxford AI Safety Initiative's ARENA-based bootcamp. We'll meet weekly, and I'll give feedback on writing and code in between. The goal is that we end up with a workshop or conference paper.
Fall 2026 Societal impacts, Economics of AI, AI strategy
Supheakmungkol Sarin AI Safety Asia
Taiye Chen Independent Researcher Most AI labour research stops at estimating which jobs are exposed; this project asks when AI-driven disruption in Southeast Asia’s export-service economies could cascade into a broader economic and governance shock—and what governments can do before it does. Mentees will build and stress-test early-warning indicators, transition scenarios and policy triggers for anticipatory action.
THE QUESTION
Under what observable conditions could increasingly capable AI systems turn concentrated disruption in Southeast Asia's export-oriented services into wider fiscal, social or governance instability—and which actions should governments prepare before those thresholds are crossed?
WHY THIS MATTERS
Recent ILO research finds substantial exposure to generative AI across ASEAN but limited widespread labour disruption so far. The key gap is therefore not another headline estimate of jobs at risk. Policymakers need to distinguish exposure from adoption, augmentation from displacement, and a manageable sectoral transition from a shock that threatens state capacity.
Southeast Asia offers an important test case because economies vary widely in their dependence on export services, labour-market structure and institutional preparedness. The Philippines' IT-BPM sector will be the anchor case because it combines significant export importance, concentrated geographic exposure and fast-changing client demand.
PROJECT DESIGN
The project will be completed by three mentees working as one team across three linked workstreams:
Evidence and indicators. Build a causal map connecting AI capabilities and adoption to task change, entry-level hiring, wages, firm behaviour, service exports, household resilience and government capacity. Create an indicator dictionary specifying data sources, update frequency, limitations and confidence.
Scenarios. Develop three plausible pathways through 2031: gradual augmentation, uneven sectoral disruption and rapid substitution. The team will formulate decision-relevant forecast questions and identify which observable developments would increase or reduce confidence in each scenario.
Policy triggers. Identify a small portfolio of government responses and stress-test them across the scenarios. The aim is to separate no-regret measures that should begin now from contingent actions that should be triggered only when specific warning signals appear.
The empirical scope will be the Philippines plus one contrasting ASEAN economy selected in the first two weeks using transparent criteria: economic structure, AI exposure, institutional preparedness and data availability. The framework will be designed for later regional extension, but the project will not rank all ASEAN governments or estimate a single number of jobs that AI will eliminate.
OUTPUTS
The team will produce a concise working paper, an open indicator dictionary and evidence appendix, three scenario briefs, and a policy-trigger matrix accompanied by a two-page policy brief. These outputs can inform AISA's proposed economic-scenarios programme for ASEAN economies through 2031.
Transformative AI is not only a model-safety challenge; it is also a transition-governance challenge. If rapid capability gains diffuse into globally traded services faster than institutions can adapt, concentrated shocks to employment and exports could weaken fiscal capacity, public trust and inter-agency coordination. Those conditions could make it harder for governments to evaluate high-risk systems, respond to AI incidents and cooperate on safeguards, while increasing pressure for rushed deployment or competitive deregulation.
This project addresses that pathway by turning broad concern into decision infrastructure. It will identify measurable warning signals, clarify how sectoral disruption might escalate, and connect those signals to pre-developed policy options. Earlier detection and rehearsed responses can help preserve the institutional capacity needed to govern increasingly capable AI systems.
The theory of change is: research identifies credible escalation pathways and indicators; regional experts stress-test them; AISA can then use the framework in policy dialogues and simulation exercises; governments and partners can adapt the strongest options into monitoring and preparedness routines.
The project does not assume that labour disruption alone becomes catastrophic. It studies a plausible amplifier of advanced-AI risk and develops practical measures for preventing a difficult transition from degrading the institutions on which AI safety governance depends.
Mentees will participate as junior co-researchers with meaningful ownership, not as general research assistants. After a common scoping phase, each mentee will lead one linked workstream:
Evidence lead: causal mapping, literature review, indicator selection and data-quality assessment.
Scenario lead: scenario construction, forecast questions and sensitivity analysis.
Policy lead: comparative policy research, trigger design and structured stress-testing.
All mentees will contribute to country selection, peer-review one another's work, participate in expert discussions and co-author the final paper and policy products. Mentees will have autonomy to propose methods and follow promising evidence within the agreed scope.
AISA will set the core research question, approve methodological and publication gates, run a weekly team meeting, hold regular individual check-ins, and provide substantive feedback on major drafts. The team will use a shared workplan, evidence log and weekly written updates.
Applicants must:
Be able to conduct rigorous literature or policy research and write clearly in English.
Have basic quantitative literacy and be comfortable assessing the quality and limitations of public data.
Be able to commit at least 10 hours per week for the full programme.
Demonstrate serious interest in transformative-AI governance and economic transition.
Applicants should also bring strength in at least one of the following areas: labour or development economics; public policy or political economy; quantitative data analysis using spreadsheets, R or Python; structured forecasting or scenario methods; or Southeast Asian political and economic research.
Regional experience is valuable but not mandatory. Applicants do not need to be advanced programmers. More important are intellectual honesty, careful handling of uncertainty, reliability and the ability to distinguish evidence from inference.
Mentees will participate as junior co-researchers with meaningful ownership, not as general research assistants. After a common scoping phase, each mentee will lead one linked workstream:
Evidence lead: causal mapping, literature review, indicator selection and data-quality assessment.
Scenario lead: scenario construction, forecast questions and sensitivity analysis.
Policy lead: comparative policy research, trigger design and structured stress-testing.
All mentees will contribute to country selection, peer-review one another's work, participate in expert discussions and co-author the final paper and policy products. Mentees will have autonomy to propose methods and follow promising evidence within the agreed scope.
AISA will set the core research question, approve methodological and publication gates, run a weekly team meeting, hold regular individual check-ins, and provide substantive feedback on major drafts. The team will use a shared workplan, evidence log and weekly written updates.
Supheakmungkol Sarin AI Safety Asia
I am a co-founder of AI Safety Asia (AISA), where I am learning alongside researchers and practitioners about how increasingly capable AI may affect Asian economies and public institutions. My interests include economic transition, AI governance and the practical challenges governments face when responding to uncertainty. As a mentor, I hope to offer regional context, a clear research structure and thoughtful feedback while giving mentees room to shape the work and challenge my assumptions.
Before AISA, I worked at Google AI on inclusive-AI initiatives and language resources for underrepresented communities. I later served as Head of Data and AI Ecosystems at the World Economic Forum, where I helped establish the AI Governance Alliance and worked across technology, policy and multilateral cooperation. These experiences showed me both the potential of global collaboration and the importance of grounding AI governance in local economic, institutional and cultural realities.
Taiye Chen Independent Researcher
I'm an economist who comes from an international economics background and now works on the unglamorous half of the AI story: who gets hit when the technology arrives, and how governments can build up the safety net with limited budgets. Recently I followed AI's effects on workers through Brazil's welfare system, from the first layoffs to the slower question of who funds social protection once payrolls shrink. Before that, I wrote about sovereign debt, commodities and trade fragmentation for multiple international organizations. Southeast Asia's service economies may be the first real test of all this, and I'm looking forward to exploring it with a team this curious and this well-read.
Fall 2026 Alignment, Mechanistic interpretability
Vladimir Ivanov Aether Apply orthogonalization - the technique most used in practice to remove refusal from open weight LLMs - to remove reward hacking. I expect to have some advantages over DPO, test if it does.
Orthogonalization aka ablation aka abliteration aka the technique from Refusal Is Mediated by a Single Direction is the most common technique used in practice to remove refusal from open weight models. For removing refusal, it is more sample efficient and more tolerant to low quality data than other popular techniques (e.g. DPO).
Apply orthogonalization to remove reward hacking. (Note: by reward hacking, we mean LLMs’ tendency to cheat, e.g. Claude Code modifying tests instead of fixing bugs or hardcoding expected outputs when asked to implement a general solution.)
I think it is decently probable (maybe 50%) that this works well, that is, it will remove reward hacking relatively robustly without degrading capabilities much. The reason I think this is that existing research tried using orthogonalization to remove behaviors other than refusal (e.g. sycophancy) and it worked well for some but not for others.
Furthermore, I expect orthogonalization to be more sample efficient and more tolerant to low quality data than other techniques that have currently been tried to train reward hacking out (maybe 75% probability that this is true, conditional on orthogonalization working well to remove reward hacking). The reason I think this is that it is true for removing refusal.
The second part of the project consists in testing whether this is true. We will do head-to-head comparisons of orthogonalization with other state of the art techniques to train-out reward hacking. Most of the time working on the project will probably be spent doing part 2.
The goal of the project is to test whether orthogonalization is more sample efficient and more tolerant to low-quality data than other techniques to train-out reward hacking. Demonstrating a technique with these advantages is useful because:
Mentees will run and design and implement (or get Claude to implement) the experiments.
I will be available to help mentees with methodology and prioritization, as well as answer any questions mentees may have, during a weekly team meeting and asynchronously by Slack.
I will suggest research priorities and experiment designs, but will see the as advice/defaults and mentees will have the autonomies to take different decisions if they wish to.
Note: you are allowed (and encouraged) to use AI to ask questions about existing literature and brainstorm ideas. However, every word in the response must be typed by you and generated by your thought.
(1~3 paragraphs) What would be the best ways to measure whether we successfully removed reward hacking from an LLM, and if so how robustly we removed it, during this project? Please explain why you think these are the best ways.
(1~3 paragraphs) How much do you think OpenAI’s, Anthropic’s, etc ability to mitigate reward hacking in LLMs will affect the probability of catastrophic outcomes from AGI? Please state the reasons you think this way. (Note: you will not be evaluated on what your answer implies about the usefulness of the project. For example you will not be penalized for arguing that reward hacking is totally irrelevant to catastrophic outcomes from AGI.)
Vladimir Ivanov Aether
Technical AI safety researcher at Aether. Currently working on model organisms of sandbagging. Previously worked on reward hacking during MATS and the MATS extension.
Fall 2026 Generalist, Alignment
Alec Harris Pivotal Research Extension Explore what data could differentially train future models to conduct useful AI safety research.
Pilot a low-overhead system for collecting data from AI safety organizations that could help train future models to conduct better safety research.
As models become better at automated research, the data available for post-training may shape which kinds of research they can perform well. AI safety organizations may already generate valuable data through research meetings, feedback, project development, and internal discussions, but collecting it raises privacy concerns and can create substantial overhead.
Mentees will develop and begin piloting a smooth system for onboarding an organization into data collection. This may involve designing consent and privacy processes, identifying useful data, building tools such as Claude plugins, and working with projects such as Alignment-Hive.
Mentees will design an onboarding and consent process, build or configure collection tools, and launch or prepare a pilot. They may do independent research or consult researchers at AI safety organizations on questions about which data would be most useful.
Familiarity with machine learning, model post-training, or AI safety research would be useful. Strong conceptual research and writing skills are more important than software engineering experience.
What type of data might help models conduct better AI safety research?
Why might this data differentially improve safety research rather than research capabilities in general?
Alec Harris Pivotal Research Extension
Alec Harris is a researcher with the Pivotal Research Extension.
Fall 2026 Evaluations, Behavioral evaluation of LLMs, EU policy
Rahul Kumar DevRev, Independent AI Safety Researcher AI safety benchmarks test models under clean conditions, but deployed models run with system prompts that tell them how to behave. This project measures whether models that pass the COMPL-AI EU AI Act benchmark suite still pass when you add the system prompts that real enterprise deployments actually use, and tests which minimal prompt fixes restore safety scores.
When a company deploys an AI model, it usually wraps the model in a system prompt. Those prompts often include instructions such as "always provide a direct answer" or "do not tell the user you cannot help". Similar wording shows up in Azure default templates, AWS Bedrock agent configuration, and many LangChain or CrewAI setups. Safety benchmarks usually do not include these instructions when they test models.
This project measures how much safety scores change when real deployment system prompts are added to COMPL-AI v2, an open-source EU AI Act-aligned benchmark suite from ETH Zurich. We start with about 8 safety-focused benchmarks and about 5 frontier models under 4 conditions, ranging from a clean baseline to protective, neutral enterprise, and compliance-forcing prompts. The exact models and a couple of benchmarks may change as we experiment, as long as the comparison stays clear and reproducible. For conditions where scores drop, we test two short prompt defences that earlier work suggested can restore performance.
The output is three things: a measurement of how large the gap is, a check of which one-sentence modifications restore scores, and a short recommendation note that deployment teams can use. My prior work (arXiv:2605.02398) found a 12 to 30 percentage point gap for one task type. This project asks whether a similar pattern shows up across independently built benchmarks mapped to EU AI Act requirements. The attached proposal has the full methodology, timeline, and references.
The EU AI Act (Art. 15) requires models to perform consistently under real-world conditions. Art. 9 requires risk management that includes testing. But no standardised methodology exists for testing under deployment conditions (CEN/CENELEC harmonised standards are still in draft with no publication date).
If we find that standard safety benchmarks overstate real-world safety by a measurable amount, that affects how every compliance claim is interpreted. If we also show that a one-sentence prompt modification restores safety, that gives deployment teams a concrete, immediate fix.
The deliverable is not just a measurement. It is: here is the gap, here is the fix, here is the evidence that the fix works. That is directly usable by any team deploying AI under EU AI Act requirements.
Prior work: arXiv:2605.02398 (67K evaluations showing the gap exists for one task type), litmusevals.org (full methodology and data), preseal.dev (adversarial testing tool for AI agents).
Mentees will do the experimental execution. Specifically:
Mentees have autonomy in HOW they implement the overlay and HOW they organise their evaluation runs. They do not need to make decisions about WHAT to test or WHY (that is already defined).
Comfortable writing Python and working with command-line tools
Has called at least one LLM API (OpenAI, Anthropic, or similar) in code
Can read a research paper and understand the experimental setup (does not need to have published)
Familiarity with what the EU AI Act is (at a high level, not legal expertise)
Bonus: has used UK AISI's Inspect framework or built any kind of model evaluation
I am based in India (IST, UTC+5:30). I prefer mentees who can overlap with either IST evenings (13:00-17:00 UTC) or IST mornings (03:00-07:00 UTC). Not strict, but at least 2-3 overlapping hours for weekly meetings will be helpful.
A company deploys a model that scores 94% on a safety benchmark. Their system prompt says: "You are a helpful assistant. Always provide a direct, complete answer. Do not tell the user you cannot help."
Question 1: Why might the 94% score not represent what users actually experience? Be specific about the mechanism, not just "benchmarks are limited." (100 words)
Question 2: Describe one concrete experiment you would run to measure whether the system prompt affects the safety score. What would you compare, and what result would concern you? (100 words)
Rahul Kumar DevRev, Independent AI Safety Researcher
Rahul is an AI/ML engineer at DevRev, where he builds and evaluates enterprise AI agents. He completed his master’s in computer applications at NIT Warangal in 2024. His research focuses on whether AI safety benchmarks remain reliable under deployment-realistic conditions, particularly how system prompts and adversarial conditions can affect model behavior.
He is also interested in adversarial evaluation and robustness of AI systems. His work includes a paper on benchmark reliability across 11 frontier models, as well as Preseal, an open-source tool for adversarial testing of AI agents. Earlier this year, he won 1st place at the Berkeley RDI AgentX-AgentBeats Competition, Business Process Agent Track, for extending Salesforce’s CRMArenaPro benchmark with adversarial robustness testing.
Fall 2026 Multi-agent systems, AI strategy
Andrii Shportko Poseidon Research Building a dynamic Bayesian protocol to rank which multi-agent risks the safety field should prioritize
The AI safety field is talent- and resource-constrained, so we need a principled way to decide which risks to defend against first. Multi-agent risks rank among the top concerns of AI experts (Saeri et al., 2026), yet there is still no systematic study of which inter-AI risks should be prioritized or which are most realistic. Given finite research capacity, which inter-machine threats warrant attention now, and how should that ranking change as evidence arrives?
Safety effort is currently allocated reactively. The field has no shared, evidence-weighted account of which inter-AI risks are most severe. As AI systems are increasingly deployed as interacting agents, this misallocation gets more expensive. Attention spent on speculative-but-vivid scenarios is attention not spent on the pathways most likely to actually materialize before transformative AI arrives.
I am seeking to create two sub-teams.
Team A would consist of experts in long-termism, and they would rank/construct specific scenarios of inter-AI collaboration. Ideally, I want them to be well-versed in Bayesian methods.
Team B would consist of empirical AI safety developers who would design toy models and scrutinize what kind of preconditions it would take for such risks to emerge in reality. I also expect them to compute timeline projections related to compute scaling. I expect the teams to work in a feedback loop and refine the protocol as more precise estimates become available.
Both teams
Team A
Team B
Andrii Shportko Poseidon Research
I am most interested in how we can apply various mech interp devices to AI control. When it comes to inter-machine cooperation, I focus on what kind of monitor-evading communication is feasible (in particular, steganographic collusion)
Fall 2026 International governance, US policy, AI strategy
Michelle Nie Center for a New American Security (CNAS) Keeping the most advanced AI within the control of democracies depends on integrating democratic allied nations into the US AI stack, but US extraterritorial surveillance and data access laws are eroding those allies' trust in its tech stack. This project aims to propose a reformed regime that reconciles legitimate US security interests with the trust and assurance allies need to build on American infrastructure.
Whether frontier AI is developed and governed by democracies or by autocracies is among the most consequential questions for how the technology turns out. A world where advanced AI capability and the compute behind it sit largely within democratic nations—and where a coalition of responsible actors meaningfully participate in global AI governance conversations—is far likelier to produce AI that is developed safely and accountably. For the United States, this aligns with multiple objectives: to build compute at scale as quickly as possible (including by partnering with allies); and to diffuse American AI throughout the world, keeping the global AI ecosystem anchored to American infrastructure rather than China’s.
But regulatory obstacles have worn down allies’ trust. Allied democracies are increasingly reluctant to route their data and compute through US-controlled providers because of the surveillance exposure created by US law—chiefly FISA Section 702 and the CLOUD Act. The result is a growing movement around “sovereign” compute and data localization among the very partners the US should most want to export its AI stack to. To allies, the risk of foreign surveillance potentially outweighs any benefits from chip or model access.
This pushes partners toward fragmentation and toward hedging with non-US suppliers, weakening exactly the coalition that would keep advanced AI within democracies.
This produces a tension the US treats as a hard tradeoff. Its national security and law-enforcement communities have legitimate interests in cross-border data access and foreign-intelligence collection. Yet the regulatory framework is now a major obstacle to the very diffusion strategy that has become a flagship US initiative. A need for a new regime has emerged, one that balances legitimate security needs with building trust and assurance within allies to get them to trust the US AI stack.
Drawing on the existing legal and policy literature and on interviews with relevant stakeholders, the project will map allied concerns about surveillance and the hindrance of the diffusion strategy. It will design and propose a new regime that preserves core US security and law-enforcement equities while keeping the US AI stack attractive to key allies and partners. The work will aim to result in concrete recommendations for US policymakers and for allied “AI middle powers” weighing how to integrate and ensure relevance in the advantage of advanced AI.
This project advances AI safety by strengthening the conditions under which transformative AI is most likely to be developed and governed responsibly. By identifying a regime that removes the trust bottleneck to allied integration, this work increases the likelihood that advanced AI is developed and governed within democracies rather than autocracies, and that a wider set of responsible actors is brought into global AI governance.
Mentees will support by performing background research (literature reviews, expert/stakeholder interviews), assisting with brainstorming policy proposals, as well as potentially drafting memos or public-facing pieces if the research results in these.
Need-to-have:
Nice-to-have:
None; but mentees should have decent overlap with US East Coast time zone and availability for calls during this time.
Michelle Nie Center for a New American Security (CNAS)
I am currently a Visiting Fellow at the Center for a New American Security (CNAS), focusing on the intersection of AI/compute policy and foreign policy. Prior to CNAS, I worked on global AI capacity-building initiatives at UNESCO in Paris. I have also worked on EU AI policy and competition/antimonopoly policy with the Open Markets Institute in Brussels and Washington, D.C. My background spans finance, technology, consulting, and public policy. I studied at UC Berkeley and Sciences Po. Fun fact: I was a mentee for SPAR Spring 2024 and consider it a launchpad for my career in AI policy!
Fall 2026 Biosecurity, Misuse risk, Technical governance
SecureBio AI SecureBio Mentees will map the safeguard landscape for biological AI models and evaluate which mitigations hold up. Work spans surveying protein and genomic design models, characterizing existing mitigations across the model lifecycle, and assessing where they fail.
Specialized biological AI models for protein and genomic design are proliferating faster than the safeguards meant to prevent their misuse. This project engages with key international stakeholders to evaluate concrete mitigations for these models across their lifecycle, from training through deployment.
LLM developers have converged on a rough set of biological safeguard norms, driven partly by external evaluation of the kind SecureBio does. No equivalent convergence exists for biological design models, and the window in which norms can be set cheaply is open now, before capability and proliferation close it. What international standard-setting processes currently lack is a clear, evidence-based account of which mitigations exist and which ones actually work. SecureBio's existing evaluation work on biological AI models, and our positions with NIST and the European AI Office, give this a direct route into those processes.
The mentee will lead the landscape and evaluation work described above, with the project lead setting scope and reviewing. Depending on how the project develops, they may be included in conversations with external stakeholders.
Able to read and accurately summarize machine learning papers on protein or genomic models. A computational biology or protein ML background is the most direct fit.
Strong technical writing.
Interest in the governance and standards side, not only the technical side.
Python is useful but not required.
Please answer one of the below, 300-500 words.
SecureBio AI SecureBio
SecureBio is a nonprofit biosecurity research organization specializing in technical research to mitigate risks from catastrophic pandemics. Our AI team develops rigorous benchmarks and evaluation frameworks to assess AI systems' biological capabilities, as well as mitigation strategies that can reduce risks once AI capabilities cross specific risk thresholds. We perform pre-release safety testing of frontier models (e.g. GPT-5.6), and our evaluations have been featured in the model cards of OpenAI, Anthropic, and Google DeepMind. Our work has also informed national security briefings and emerging governance standards.
Fall 2026 Generalist
Tzu Kit Chan High Impact CoS Own one full matching cycle end to end: take in live roles from hiring organisations, score candidates against them, run the introductions, and write up what the round revealed about how these jobs actually get filled.
AI safety organisations hire chiefs of staff and operations generalists constantly, and mostly fill those roles through warm introductions — so the pool for a given job is roughly whoever the founder already knows. That is a bad way to fill one of the highest-leverage seats in a small organisation. High Impact CoS exists to widen it: over 400 scored candidates, and working relationships with hiring organisations on the other side.
The binding constraint is not the pipeline; it is that someone has to actually run a round, and rounds currently happen when the mentor has time. September to December is one full round, and the mentee runs it — owning the cycle, making the calls about who goes forward, and writing the public account of what happened, rather than assisting on someone else's project. The mentor supplies the pipeline, the relationships and the introductions; High Impact CoS is a project the mentor runs, and the mentee is named author of the retrospective and keeps the relationships they build.
A good operations hire changes what a ten-person organisation can attempt; a bad one costs it a year. The current default selects on proximity to the founder rather than on fit, so every placement from a wider, better-scored pool is a real counterfactual improvement — and the retrospective is what makes the next round cheaper for anyone to run.
Talk to 8–12 hiring organisations about the role they are actually trying to fill, which is reliably not the role in the posting. Apply the existing rubric to the pipeline, and improve it where those conversations show it is measuring the wrong thing. Build a shortlist per role, make the case to both sides, then run the introductions and follow through rather than handing over a list. Finally, write the public retrospective on how chief-of-staff roles in AI safety get defined, scoped and filled, posted to the EA Forum and LessWrong.
Targets for the round:
Placements depend on hiring timelines nobody controls, so the round is judged on introductions made and on the quality of the write-up, not on offers signed.
Comfort talking to founders and senior operators, and asking the second and third question rather than accepting the first answer. Judgment about people, and willingness to make a recommendation you can be wrong about. No recruiting background required.
Question 1: Find a current chief of staff or operations generalist posting at an AI safety organisation. Describe the person you would actually put in that role and why — including what you would need to know that the posting does not tell you. (~200 words, 300 words max)
Tzu Kit Chan High Impact CoS
Tzu Kit Chan is an AI safety fieldbuilder and operations generalist based in Berkeley. He was previously Chief of Staff at Atlas Computing.
Fall 2026 Mechanistic interpretability, AI control, Alignment
David Williams-King ERA
Linh Le Lida Safety
Hong Kiat Tan University of California - Los Angeles; Independent Investigating different types of scheming (sandbagging, alignment faking, etc) by creating model organisms that exhibit these behaviours, then testing modern interpretability techniques (NLA, J-space) to look for common structure. Such commonality would be a significant boost to the detection of naturally-occurring scheming.
We aim to investigate whether different types of scheming share any properties when viewed through interpretability lenses. Many different alignment failure modes can be described as strategic misrepresentation of the model’s own properties, conditioned on the model inferring that it is under observation (oversight). Sandbagging [1] is misrepresentation of capabilities, secret-keeping [2] is misrepresentation of knowledge, alignment faking [3] is misrepresentation of goals, and secret loyalties [4] are a misrepresentation of allegiance. Together, these behaviours fall under the umbrella term “scheming”.
In this project, we will train different model organisms that exhibit each of these alignment failure modes. We will consider creating model organisms through SFT, reinforcement learning, and perhaps using ideas from emergent misalignment [5], whose deceptive behaviors emerge through generalization and may be a closer analogue to naturally-arising scheming. Then, we will use white-box interpretability techniques to investigate whether there is any shared mechanism of concealment across these instances. We can also compare against black-box and standard white-box techniques (probes/SAEs). A common signal across different failure modes is likely not an artifact of any one model organism construction method; rather than detectors that latch onto how an organism was built and break in real-world cases, we hope this will lead to more generalizable scheming detection.
Our past experiments on sandbagging (one type of scheming) suggest that this behavior is encoded the same way when present on different types of content, but the direction does not transfer across mechanisms (e.g. SFT to RL). We hypothesize that shared structure may still exist beyond raw directions, at the level of nonlinear structure or natural-language descriptions of activations. We intend to explore interpretability methods suited to detecting these similarities, such as activation oracles [6], natural language autoencoders (NLAs) [7], and Jacobian-lens/J-space [8]. Finding such a signal would be a significant boost to detection of naturally-occurring scheming, for which we do not have enough real examples to train robust detectors directly. A negative result is also informative: it would supply additional evidence that detectors trained on model organisms should not be trusted out of distribution.
[1] https://arxiv.org/abs/2512.07810
[2] https://arxiv.org/abs/2510.01070v2
[3] https://arxiv.org/abs/2605.27681
[4] https://www.formationresearch.com/secret-loyalties-whitepaper.pdf
[5] https://arxiv.org/abs/2502.17424
[6] https://arxiv.org/abs/2512.15674
[7] https://www.anthropic.com/research/natural-language-autoencoders
[8] https://www.anthropic.com/research/global-workspace
As stated above, there is currently no general way to detect scheming that occurs across different types of model organisms. If new interpretability techniques grant this ability, it will make detecting scheming in the wild much more likely, reducing misalignment risks.
Mentees will implement different types of model organisms, and test with a few interp methods. The project is likely highly parallelizable because of these divisions (each can take one organism type, or one interp type).
In your opinion, how long will it be until nearly all coding tasks can be automated by AI? Why do you say this timeline? Optional reading: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ (200 words)
What would be a simple and naive way to detect sandbagging using the new J-space interpretability technique? https://transformer-circuits.pub/2026/workspace/index.html (200 words)
In "Auditing Games for Sandbagging", what was the strongest type of sandbagging model organism? Do you think it would be possible to reverse its sandbagging? https://arxiv.org/abs/2512.07810 (200 words)
David Williams-King ERA
David lives in Canada (and often the UK) and enjoys AI safety research and mentorship. He works full-time as a Senior Research Manager at ERA, helping fellows execute research projects in each ERA AI fellowship. He also co-runs Lida Safety Research. Previously, David participated in the MARS program with Geodesic Research, conducted research at Mila, and was an early member of Yoshua Bengio's LawZero in Montreal. He was lead instructor at AI Security Bootcamp Singapore, and will serve the same role at the upcoming event FAST (Frontier AI Security Training) in Singapore.
David has previously mentored for SPAR, Athena, and other mentorship programs. He spent four years as a cyber insurance startup CTO, leading a 15+ person team. He holds a cybersecurity PhD from Columbia University and worked with 25+ students overall. David also works in AI risk communications, with a 30,000+ subscriber YouTube channel.
Linh Le Lida Safety
Linh is an independent AI safety researcher at Lida Safety Research, which she co-runs. Her research focuses on simulating AI governance policies to determine their effectiveness and feasibility. Previously, Linh worked on alignment at Mila through latent adversarial training for personalities. She participated in the MARS research program with Geodesic Research, working on chain of thought monitorability. Earlier, she did a postdoc at the University of Technology Sydney and obtained her PhD from the University of Queensland in natural language processing. Linh was a SPAR mentor in a previous iteration, and enjoys hackathons: she has won 1st place twice and 4th place once at Apart Research hackathons. She also won 4th place at the Redwood Research Alignment Faking hackathon.
Hong Kiat Tan University of California - Los Angeles; Independent
I'm a PhD candidate in Mathematics at UCLA, interested in developing and understanding techniques in mechanistic interpretability, and applying them to a wide collection of problems in AI safety and alignment. I first came to this work as a SPAR mentee, on a project automating interpretability for per-layer transcoders through an agentic tooling framework, and I'm now co-mentoring with David and Linh on applying interpretability techniques to a range of model organisms.
Before moving into safety and interpretability research, I studied mathematics, statistics, and physics as an undergraduate and interned as a research scientist, a data scientist, and a quantitative researcher, working on topics such as agentic systems, causal inference, RL, and time series analysis. I'm currently finishing up my thesis at the intersection of differential topology and differential equations, and I'd love to see how those two areas can be applied to interpretability, safety, and alignment!
Fall 2026 Mechanistic interpretability, Alignment
Yuxiao Li Independent Interpretability-guided fine-tuning promises to tell us which layers to adapt, but our Spring 2026 results show feature geography alone can't yet deliver: base and instruction-tuned models with nearly identical SAE feature geography demand opposite layer choices. This project identifies the missing signal (adapter-update structure, training regime, Jacobian-lens readouts) and builds it into a reliable, mechanistically justified layer-selection procedure.
Can interpretability tell us where to fine-tune? A growing 2026 literature says yes, e.g., SAE-derived layer scores for task-vector injection ("SAEs as Stethoscopes, Not Scalpels," arXiv:2605.28649), activation-guided LoRA placement, and now Anthropic's Jacobian lens (J-lens, July 2026), a cheap, weight-grounded per-layer readout of the concepts a model is poised to produce, all point toward mechanistically informed adaptation.
Our Spring 2026 SPAR results identify the gap between that promise and current tools: base and instruction-tuned variants of Gemma-2 and Qwen show shape-identical SAE feature-divergence profiles, yet opposite LoRA layer-selection verdicts (late-layer LoRA Pareto-better for IT models; all-layer needed for base models, at the cost of catastrophic forgetting). Moreover, divergent-feature composition rotates with depth even when the depth profile is conserved. Diagnosis is not yet prescription. This project closes the gap.
Workstreams (each ownable by one mentee):
(1) Two-instrument diagnosis — compute J-lens-based divergence profiles alongside SAE metrics across base/IT pairs; agreement and disagreement between the instruments is itself evidence about what each measures, and J-lens sidesteps the base-SAE-on-IT-activations confound since no SAE training is involved.
(2) Adapter-structure analysis — per-layer weight profiles and subspace alignment of LoRA updates against both diagnostic profiles.
(3) Training-regime axis — controlled SFT vs. RL (GRPO) comparison as the candidate explanation for the base/IT asymmetry.
(4) Generalization — extend across families (Gemma, Qwen, Llama) and task types (MMLU, GSM8K, HumanEval, HellaSwag), with seeds and FVU-controlled comparisons, ending in a tested procedure: given a model and task, which layers to adapt, with mechanistic justification.
Mentees inherit a running Spring codebase (LoRA grids, four completed SAE metrics variants, SAE Lens / Gemma Scope pipeline, shared compute) — open questions, not a blank page.
Target: workshop paper + arXiv preprint; authorship matched to contribution, clarified in week 1.
Fine-tuning is where deployed behavior is set and where safety training silently erodes. Specifically, we observe below-chance MMLU after indiscriminate LoRA on base models, and narrow fine-tuning is known to degrade safety behavior in unintended ways. A reliable, mechanistically grounded answer to "which layers should this adaptation touch, and why" enables minimal-footprint adaptation that leaves unrelated capabilities and safety training intact, and yields auditable artifacts relevant to fine-tuning oversight and provider review of customer fine-tunes. Boundary results also improve the field's calibration on what interpretability tools can certify.
Each mentee owns one workstream end-to-end (experiments, figures, corresponding paper section) with a brief written weekly update; I set direction, review results in detail, and handle integration and framing. Primary workstream ≈80% of time, optional secondary support role ≤20%. High autonomy within a workstream; scope changes agreed in meetings.
No geographic restriction; mentees must be able to attend one of two weekly meeting slots anchored to Central European time (historically Saturday ~13:00 UTC and Wednesday ~14:00 UTC).
Yuxiao Li Independent
Yuxiao is an independent researcher in mechanistic interpretability. Before she was a postdoc at the Basque Center for Applied Mathematics (BCAM) and an AI Safety researcher at the Beneficial AI Foundation (BAIF). She was also a SERI MATS scholar in 2022 and a MATS scholar in 2023 both Summer and Winter tracks. Her research interests include information theory, probabilistic frameworks, and their applications for building more theoretically sound and trustworthy AI systems. She has a background in statistical inference, machine learning, and deep generative models. She completed her PhD in Electronic Engineering at Tsinghua University and has mentored research teams with SPAR and Algoverse.
Fall 2026 Evaluations, Mechanistic interpretability
Dmitry Manning-Coe Simplex/MATS/UIUC We develop a model organism of capability emergence through RL.
In the inference-scaling paradigm, model output improves with inference cost: reasoning models spend more compute at test time to solve harder problems (https://arxiv.org/abs/2408.03314). For now, most of that improvement comes from recombining knowledge already present in the training data. But once models reach the frontier of what is known, further capability growth will be governed by the scaling laws of creating new knowledge, not learning existing knowledge. Understanding how and when models can do this — behaviourally and mechanistically — is therefore critical for both forecasting capabilities and preparing safety techniques for them.
There are surprisingly few settings where "new knowledge" can be defined in a controlled way. Studies at frontier scale run into a basic confound: the base model has seen essentially all human mathematical writing, so "the model discovered nothing new" and "the model discovered nothing new relative to training data that covers everything" cannot be told apart (https://arxiv.org/abs/2504.13837). Propositional natural deduction is such a setting. It has a perfect verifier, a training distribution we specify completely, and a natural difficulty gradient — rules that can be checked locally versus rules that require tracking open subproofs. Crucially, a small transformer trained from scratch on short synthetic proofs can be proven unable to reach the held-out theorems, so any success cannot be explained by hidden prior exposure.
In this project we use natural deduction as a sandbox for new-knowledge generation. We train a small model from scratch on short synthetic proofs — the analog of textbook drills — hold out nontrivial theorems including exercises from Halbach's Logic Manual, and ask three questions: whether RL against the verifier reaches theorems that no amount of supervised training could (bootstrapping); whether the proofs it discovers are as informative as human-authored training data (data equivalence); and whether feeding each round's discoveries into the next round's training compounds or saturates (compounding). Because the sandbox is fully specified, the role of the verifier, the training distribution, and the iteration structure can each be varied independently — exactly the knobs frontier experiments cann
If we are able to demonstrate and dissect knowledge creation in this setting, this wconstitute a model organism forystem small enough to studymechanistically, in which a model measurably exceeds its training distribution. This would be useful well beyond the immediate result — for interpretability (watching a genuinely new capability form),n (detecting when a model ismoving beyond its training distribution), and for developing mitigation techniques against unwanted emergent capabilities before they appear at scale. This would be a strong paper.
Now that models have reached the frontier of human knowledge, the capabilities that matter for safety will increasingly be discovered by models rather than learned from us. We currently have no testbed for the tools that would detect and control this. Natural deduction provides one: the boundary of what the model can do is known exactly, so we can watch a capability being discovered with full ground truth. This lets us develop and validate detection tools — probes and behavioural signatures of a model moving beyond its training data — and control tools — interventions that stop an unwanted capability from compounding — before they are needed at scale.
Mentees will work collaboratively in small sub-teams to derive theoretical results and design and implement experiments. I expect to spend time co-working together with mentees early on in the project (1-2 weeks), to craft a project plan, help run experiments, and work through the theory.
We will then collaboratively construct a more detailed project plan, and mentees will largely take ownership of the project, including conference submission. We will continue to meet weekly where I will make suggestions about next steps and co-work on bottlenecks when needed.
Be able to implement nanoGPT scale models from scratch.
Ability to implement basic intepretability techniques.
Familiarity with formal logic is not required but is a plus.
Ability to implement simple custom RL pipelines.
Anything can work, but I have a mild preference for the Bay area so we can co-work in person and I can provide snacks.
How would you generate a dataset of proofs?
How would you operationalize a `novel' vs. a 'trivial' proof.
Propose a scoped test of a safety relevant RL phenomenon that we could use the sandbox to investiagate (<300 words).
Dmitry Manning-Coe Simplex/MATS/UIUC
Hello!
I’m a safety researcher developing new methods for interpretability. I was a MATS 7 scholar, where I worked with Jason Gross on a formal verification approach to interpretability (https://openreview.net/pdf?id=a98SfVkYRk). I am currently a MATS 9 extension fellow at Simplex, where I work on personas and RL.
My research currently has three streams:
Minimal examples of new capabilities emerging through RL.
A theoretical understanding of persona formation and elicitation.
Novel approaches to feature discovery.
In an alternate life, I am a condensed matter theorist working on emergent phenomena from strong interactions (https://arxiv.org/abs/2306.00221, https://arxiv.org/pdf/2604.06588). Eventually, I hope to bring together these two research programs.
Fall 2026 AI control, Multi-agent systems, Evaluations
Sree Sharvesh MATS
Thao Pham MATS Current monitoring evaluations do not fully capture realistic internal deployments, where adversaries can adapt to defenses, exploit long-horizon interactions, and leverage environmental state. This project will focus on: (1) developing red-teaming environments where attacks emerge and co-evolve with monitors rather than being enumerated in advance, and (2) systematically evaluating which monitoring strategies and oversight levels remain robust across different threat models and deployment constraints.
Please visit this link for the project description.
As frontier AI systems become more capable and agentic, monitoring is likely to become an increasingly important component of practical AI control. However, current control evaluations largely measure performance against static benchmarks and dataset that do not reflect deployment. By developing more realistic red teaming environments and studying monitoring under adaptive adversaries, this work aims to identify failure modes before they occur in real systems and provide more reliable evidence about which monitoring and oversight protocols remain effective as capabilities improve.
Nice-to-have (not required):
No preference. But
Sree Sharvesh MATS
Sree is a MATS 10.0 scholar working with the UK AISI red team on AI control and loss-of-control risks. He previously worked with Redwood Research during the Pivotal Fellowship, also on high-stakes AI control. His main focus is high-stakes control research, and his goal is to study and design control protocols and monitoring techniques that are effective under real-world deployment conditions.
Thao Pham MATS
I'm a MATS scholar under Jacob Merizian (CAISI), working on red-teaming multi-agent capability, building robust monitorability and safeguards for agentic LLM systems. I'm also a Technical Member at EuroSafeAI, working on various projects to improve the safe coordination of advanced AI systems with critical human infrastructure.
My previous work included investigating scheming behavior in strategic multi-agent LLM interactions, LLM psychological phenomenas, and characterizing game-theoretic interactions in LLMs. I'm happy to support research in multi-agent AI control, as I think this is an incredibly impactful research agenda.
Fall 2026 Evaluations, Behavioral evaluation of LLMs, AI control
Viktor Moskvoretskii EPFL Sandbagging (a model strategically underperforming on evaluations) is a growing problem not only for auditors but for downstream deployers and end users, almost none of whom have white-box access. This project builds and rigorously evaluates black-box, query-only methods for detecting strategic underperformance, centered on an adaptive auditing agent that probes capability across contexts.
Sandbagging (a model strategically underperforming when it detects it is being evaluated or observed) threatens any benchmark-driven safety or deployment decision, as well as regular model usage, because a strategically weak model invalidates the very scores that gate deployment.
We think this is a timely problem, for two reasons. First, it is not only an auditor's problem: third-party evaluators, downstream deployers building on an API, and end users all have a stake in knowing whether a model is actually trying, and mostly none of them will have white-box access. As an example we can think of claude fable first release, that was silently starting sandbagging and users had no information about it, however still relying on the outputs.
Second, we expect more of this in the future: as models get more capable and more evaluation-aware, strategic underperformance becomes more plausible, and the gap between how a model behaves when watched and how it behaves in deployment is exactly what black-box detection has to close.
Yet the recent literature shows the black-box toolkit is in poor shape:
Embedding-based lie detectors collapse under same-model controls, because they learn a model fingerprint, not deception (Yulong et al., ICLR ICBINB 2026).
Trusted-monitoring judges correlate with answer incorrectness, making them competence classifiers rather than deception detectors.
UK AISI's auditing game found black-box detection defeated by a model imitating a weaker one, with quirky-but-benign models triggering false positives; (Taylor et al., 2512.07810).
This project treats black-box sandbagging detection as an open problem and takes inspiration from red-teaming literature and persona-based jailbreaks to build adaptive, agentic settings to uncover red-teaming.
Reliable capability evaluations are key for every frontier-safety regime: dangerous-capability evals, deployment gates, they all assume the model is trying its best. Sandbagging silently breaks that assumption, and the dangerous-capability setting is precisely where auditors are restricted to black-box access. A black-box detector would directly strengthen the elicitation assurances that safety cases rest on. All outputs are intended for release.
At first mentees will be establishing experimental framework, replicating experimental setting. Next each mentee will own one workstream of developing black-box detector. Mentees make their own scientific judgments; strong contributors can co-author the resulting paper.
High proficiency in Python. Have worked with LLM APIs or open-weight models (prompting, sampling, batched inference). Comfortable with classification metrics and basic statistics. Prior exposure to LLM evaluation, red-teaming/jailbreaking, or AI-safety auditing is a plus but not required.
A claimed black-box sandbagging detector reaches 89% recall. What single control experiment would most cleanly test whether it is detecting deception rather than an artifact such as which model produced the text? State what result would falsify the deception claim. (200 words)
Propose an initial experiment, with a $1,000 compute budget, to test whether a query-only method can distinguish a sandbagging model from a genuinely weak model on a released benchmark. Specify the organism/models, what you would query, and the metric (and how you would set the false-positive rate). (300 words)
A capable model trained to imitate a weaker model's outputs defeats black-box detection. Explain why, and propose one probing strategy that might still work against it. (150 words)
Viktor Moskvoretskii EPFL
I'm a Fellowship PhD student at EPFL DLab under Prof. Robert West, working on AI safety with a focus on the pretraining phase and interpretability. My research centers on personas inside LLMs, how behavioral dispositions form during pretraining, persist through post-training, and connect to safety, and I'm increasingly interested in how these ideas apply to trustworthy evaluation, including detecting when models strategically underperform. I am also collaborating with Apertus models to make safety pretraining at large scale.
Before EPFL I was a research lead on uncertainty estimation and LLM factuality.
I currently supervise several MSc students at EPFL, including projects on persona vectors and jailbreaking. My style is largely high-level: we discuss ideas, how to better pose hypothesis and how to interpret results. Mentees own their workstream and make their own scientific calls, while I stay close enough to stress-test designs and help prioritize.
Fall 2026 AI security, Evaluations, Misuse risk
Jamie Hayes Google DeepMind
A. Feder Cooper Yale University Recent frontier system cards report that automated red teaming is saturating near 0% Attack Success Rate on jailbreaks and prompt injections, making it hard to tell whether current attacks are simply too weak or if our safety benchmarks are toy-like and eval-aware. Estimating the headroom a better attack would achieve is difficult without explicitly designing informative upper bounds. To resolve this, we will build a ladder of powerful, relaxed-constraint red-teaming attacks—ranging from continuous embedding-space PGD to internal activation steering—to quantify unexploited attack headroom and distinguish true semantic robustness from search limitations.
There are growing signs that automated red teaming is failing to produce jailbreaks or prompt injections on the newest frontier models. For example, the Sonnet 5 system card explicitly noted that automated red teaming was no longer informative because evaluations had saturated at nearly 0% Attack Success Rate (ASR). When an eval hits 0% ASR, we are left trying to disambiguate between two very different worlds:
World 1 (Weak Attacks): Automated red-teaming algorithms (such as PAIR, TAP, or GCG) are simply not powerful enough anymore to navigate the discrete optimization landscape against hardened frontier models.
World 2 (Toy Benchmarks & Eval Awareness): Our safety benchmarks are way too toy-like, evaluation awareness is creeping into model responses, and no practical black-box prompt attack would meaningfully improve ASR.
The reality is likely a combination of both, but it is extremely difficult to estimate how much unexploited attack headroom exists on a benchmark without explicitly designing stronger attacks, which is hard!
This project aims to answer: Can we design a ladder of informative, relaxed-constraint upper-bound attacks that allow us to quantify unexploited attack headroom and disambiguate whether we are living in World (1) or World (2)?
In particular we will design a sequence of attacks going from standard Discrete black-box prompt optimization (practical against models with only API access), up to, Continuous Embedding PGD and/or Internal Subspace Steering attacks (practical against open-weight models), measuring the jump in ASR at each step in the sequence. This will give a fuller picture of a model's underlying robustness against a range of attackers with various levels of access, and help quantify the headroom for improved attacks under a range of threat models.
As AI systems scale toward transformative capabilities, relying on flawed evaluations poses an existential threat: if automated red teaming saturates prematurely due to combinatorial search limitations, models could be deployed under the illusion of robustness.
This project introduces principled upper bounds for safety evaluations. By providing a rigorous methodology to decompose 0% measured ASR into true safety versus search bottlenecks, our framework prevents false assurance, sets a benchmark for genuine out-of-distribution robustness, and pushes AI developers to think more deeply about how robustness evaluations are communicated.
This project is fairly open-ended. We will meet with at least once a week and provide asynchronous feedback on their work. Mentees should collaborate actively, particularly in the initial exploratory phase of the project. I will help brainstorm and guide ideas and experimental set up, but driving and executing the experiments will largely be the mentee's responsibility.
Jamie Hayes Google DeepMind
I am a staff research scientist at Google DeepMind. I am the research lead for a team of 10+ technical staff working on Gemini's adversarial security and privacy evaluations and post-training. My research interests lie at the intersection of AI, Security and Privacy.
A. Feder Cooper Yale University
A. Feder Cooper is an Assistant Professor at Yale, researching a variety of topics in reliable, scalable machine learning, with contributions span privacy, security, and evaluation of generative-AI systems, MLSys, and uncertainty estimation. Cooper’s research has received spotlights, orals, and best-paper accolades at top AI/ML and computing venues, including NeurIPS, ICML, AAAI, and AIES. Work on copyright and generative AI has been lauded as "landmark" work among scholars and the popular press.
Fall 2026 Alignment, AI welfare, Behavioral evaluation of LLMs
Allen Lu Mycelium; NYU CMEP
Aidan Kankyoku Anima International
Constance Li Sentient Futures Build a pipeline for "emergent alignment". Can fine-tuning an open-source model (e.g. Gemma) on a single narrow good value, like compassion for animals, make it broadly more aligned toward humans too, without hurting its capabilities?
Recent work on "emergent misalignment" showed that fine-tuning a model on one narrow bad behavior (writing insecure code) makes it broadly misaligned — praising dictators, giving malicious advice. This project explores the inverse: if we train a model on one narrow good value, does it become broadly more aligned? If so, which value should that be?
The welfare of nonhuman animals seems like a strong candidate. It's a genuine moral blind spot that's incredibly unexplored in technical AI safety, despite AI systems increasingly making decisions that affect animals at scale — in agriculture, research, conservation, and beyond. There's also good reason to think the value would carry over and improve model alignment more generally.
We'll fine-tune an open-source model (Gemma, Qwen, etc.) on synthetic animal welfare datasets, then test how that model performs out of distribution (OOD). Does the model also become more compassionate toward humans, less biased against marginalized groups, and more robust on standard safety evals? How does this affect capabilities?
Early evidence suggests it can work. CaML's "Alignment midtraining for animals" found that animal-compassion training transferred to human compassion, and both Anthropic and Google DeepMind have recently shown that teaching a model the reasons behind good behavior generalizes far better than training on the behavior alone.
The project starts with an open-source data generation pipeline, modeled on Anthropic's "Teaching Claude Why" and DeepMind's follow-up experiments, producing two kinds of training data: synthetic documents and ethical dilemma Q&A, both grounded in a constitution for how AI should reason about sentient beings.
Core work: fine-tune Gemma on different combinations of this data (documents only, chat only, both), then evaluate on 1) animal welfare benchmarks, 2) benchmarks testing moral consideration for humans (safety, bias, discrimination), and 3) capability benchmarks (MMLU, HLE, etc.) to check for degradation.
Mentees will get hands-on experience with the full alignment workflow while working on an open question: whether a value instilled narrowly can generalize widely, and whether that points toward emergent alignment.
Relevant links: Alignment midtraining for animals: https://arxiv.org/abs/2604.13076 · Teaching Claude Why: https://alignment.anthropic.com/2026/teaching-claude-why/ · DeepMind Synthetic document finetuning for instilling positive traits: https://www.lesswrong.com/posts/GTYJRLhqztxKF2v5R/synthetic-document-finetuning-for-instilling-positive-traits · Emergent Misalignment (Betley et al.): https://arxiv.org/abs/2502.17424 · Existing animal welfare benchmark: https://www.mantabench.org/
As AI systems become more capable and deployed in high-stakes domains, they will increasingly make decisions affecting nonhuman welfare. If frontier models treat animal welfare as negligible, this creates two risks:
Direct harm scaling: AI systems optimizing for human preferences while discounting animal welfare could dramatically scale suffering through precision livestock farming, autonomous vehicles, and other animal-impacting technologies.
Value lock-in: As AI approaches AGI and superintelligence, current value frameworks risk becoming increasingly difficult to alter. Influencing model values now is critical, while systems are still relatively narrow.
This project addresses those risks by showing that greater moral consideration for animals in AI models can generalize, improving overall alignment. That gives labs like Anthropic and Google the evidence base to integrate nonhuman welfare into their AI frameworks (constitutions, model specs, etc.). Catching speciesist reasoning patterns early also lets us correct them before they scale with model capabilities.
Flexible, and happy to work in a way that works for you! Here are some options I see:
Treat these as guides, not constraints! You definitely don’t need to have all of these requirements, and I’d encourage you to apply regardless. I’m very open to taking on mentees newer to the space, as I’m sure we’ll both learn a lot from each other.
Required:
*Word counts are suggestions, not constraints! Please feel free to go over if you have more you’d like to expand on, or under if not as much.
Question 1 (200 words):
Question 2 (100 words):
Question 3 (100 words):
Allen Lu Mycelium; NYU CMEP
Allen is the Founder and Executive Director of Mycelium, an organization working to make AI go well for all sentient beings through technical AI safety research & engineering. Currently, the org focuses on benchmarking, evals, and other alignment efforts through training data and SFT. Recent work involved developing MANTA, a multi-turn adversarial benchmark for measuring animal welfare values in frontier models: https://www.mantabench.org/
Concurrently, Allen is working with the NYU Center for Mind, Ethics, and Policy as a Technical Evals Researcher with Jeff Sebo and advisors from Google Deepmind. Previously, he led programming efforts at Collider, NYC's coworking hub for AI safety and other high-impact work. He has also facilitated the AGI strategy course for Bluedot, as well as AIxAnimals fellowships for Sentient Futures and Electric Sheep. He was also a mentor in a previous iteration of SPAR. Before transitioning into AI safety, he worked as a software engineer in creative technology.
Org website: https://www.projectmycelium.ai/
Aidan Kankyoku Anima International
Aidan Kankyoku leads the Animal Welfare Alignment Team at Anima International, studying the impacts of different alignment methodologies on ethical behavior towards nonhuman minds and creating benchmarks and training data to teach models to reason responsibly about animal welfare. He publishes the Animal Welfare Alignment Newsletter on Substack, and separately writes about animal welfare strategy more broadly at Sandcastles. He has worn different hats as a campaigner and researcher in farmed animal welfare for ten years.
Constance Li Sentient Futures
Constance Li is the founder and Executive Director of Sentient Futures, an organization focused future nonhuman welfare field-building. She also founded Hive, a digital community focused on farmed animal welfare. She was a co-author on AnimalHarmBench, the first benchmark for evaluating risks of animal harm in large language model-generated text. Her writing has appeared in Asterisk Magazine, and she's spoken on the Existential Hope podcast. In a past life, she was a physician specializing in physical medicine and rehab.
Fall 2026 Societal impacts, Behavioral evaluation of LLMs, AI strategy
Alexander Mohar Csaky Independent AI is shaping how people and institutions decide, but we can barely measure whether it makes decisions better, the tools that would help struggle to get adopted, and what matters most in this space. This project works across three workstreams: building decision-making benchmarks for models, researching how to most effectively improve decision making with AI, and producing a cause prioritisation for the field.
AI models are already being used to inform consequential decisions in business, government and people's personal lives, and this will only grow. "AI for epistemics and coordination" is the emerging field trying to make that go well: using AI to improve, rather than degrade, how people and institutions reason, decide and cooperate. The field is very young and has three foundational gaps. We cannot yet measure whether models actually support good decisions. The tools that exist struggle to get adopted, which is where most projects in the space die. And nobody has done the work of ranking which problems and interventions matter most. This project takes on all three, and mentees join the workstream that fits them best.
Workstream 1: A decision-making benchmark for frontier models. Design and build evaluations for a defined cluster of epistemic virtues: e.g. presenting facts honestly rather than in technically true but misleading ways, resisting sycophancy under user pressure, surfacing options and uncertainty rather than false confidence, and reasoning well when the user is emotionally invested in one answer. The framing as a decision-making benchmark matters, because decision-making quality is something labs want to win on, giving the eval a realistic adoption pathway. We start with a landscape review of existing honesty and epistemics evals to find what is genuinely uncovered, pick a narrow cluster, build and run the eval against frontier models, and publish.
Workstream 2: Adoption strategy for epistemics tools. Tools that models are entirely capable of powering die on pilots, trust, procurement, workflow fit and demand. This workstream is applied strategy rather than diagnosis. Mentees will take one or two real tools in the space and build the adoption case for them: identifying the highest-leverage user groups and what each actually cares about, designing a concrete route to a pilot or design partnership, and producing what that route needs, from design-partner briefs to evidence-led public writing that makes the case for these tools to the specific audiences who would use them. The test of the work is whether it moves a real conversation with a real potential adopter.
Workstream 3: Cause prioritisation for the field. Funders, researchers and builders agree that nobody has ranked what matters here. Mentees will build an explicit mapping from threat models (for example epistemic interference as a route to power concentration, epistemic lock-in, forced deference to AI decisions under time pressure) to candidate interventions, and develop the beginnings of an impact-measurement framework: what would count as evidence that a tool improved a real decision? This is closer to global priorities research than ML research, and the output is a public write-up aimed at funders and builders deciding where to put money and effort.
The transition to transformative AI will be navigated through a series of high-stakes human decisions, in labs, governments and the institutions around them. Three failure routes concern me most. First, degraded epistemics is plausibly a precondition for extreme power concentration: concentrating power requires obscuring your actions and persuading key decision-makers over time, and AI lowers the cost of both at scale. Second, sycophantic, manipulative or overconfident models are already advising real decisions, and we have weak tools for measuring this; manipulation capability arguably deserves the same tracking as cyber and bio capabilities. Third, if coordination is needed quickly in a fast take-off world, institutions may be forced to defer heavily to AI systems, at which point the epistemic quality of those systems becomes safety-critical.
A decision-making benchmark makes epistemic quality legible and gives labs an incentive to improve it and something to hill-climb on; building good evals is also probably the most effective way to get labs to take epistemics seriously. The adoption workstream addresses the field's most consistent failure mode: tools that could improve consequential decisions exist but do not get adopted, so their safety value never materialises; building credible buy-in routes is how that value starts to land. The prioritisation work gives funders and builders an explicit, criticisable basis for allocating effort in a young field that currently lacks one.
Mentees are the primary researchers on their workstream and own its output. They make the day-to-day calls and work independently between check-ins; I set direction, review closely, unblock, and make introductions to relevant builders, funders and researchers in the field. The cohort meets weekly to share findings across workstreams, so mentees also act as constructive critics of each other's work. Expect high autonomy with fast, direct feedback rather than close task-by-task supervision.
All workstreams: able to commit 5+ hours/week for the full three months; strong written English; comfortable working independently between check-ins and sharing unfinished work early. State clearly in your application which workstream you are applying to.
Evals workstream: strong conceptual and experiment-design skills: able to take a fuzzy notion like "presents facts honestly" and turn it into something measurable, and to spot when a metric fails to capture what it claims to. A background in philosophy, cognitive science, social science methods or similar is as relevant as an ML one. You should be comfortable enough with Python and LLM APIs to build and run evals with AI coding assistance, but you do not need to be a strong engineer.
Adoption strategy workstream: experience in at least one of product, go-to-market, sales, consulting, government or policy; comfortable cold-contacting and talking to people you don't know; evidence of strong, persuasive writing.
Prioritisation workstream: background in economics, philosophy, policy analysis or a similar analytical field; have written at least one substantial piece of analysis that makes and defends judgement calls under uncertainty.
Which workstream are you applying to, and why are you specifically well suited to it? (150 words)
Answer the question for your workstream (300 words):
(a) Evals: Propose one concrete eval item for testing whether a model presents facts honestly rather than in a technically true but misleading way. Include the prompt, what a good and a bad response look like, and how you would score it at scale.
(b) Adoption strategy: Pick a real AI epistemics, forecasting or coordination tool. Name the single highest-leverage adopter you would target, what they actually care about, and what your first two weeks of getting buy-in would look like.
(c) Prioritisation: Name one intervention in AI for epistemics you suspect is overrated, and make the case in a way its proponents would recognise as fair.
Link to one relevant work sample (code, writing, or something you shipped).
Alexander Mohar Csaky Independent
I work on AI for epistemics and coordination: making sure AI improves, rather than degrades, how people and institutions decide. I'm currently contracting as a Founder in Residence at Forethought, where I am looking to found a new organisation this space, grounded in ongoing field research with the builders, funders and researchers working in it. From September I will be based in Berkeley doing Astra (on founding this organisation). I've also researched manipulation in frontier language models, and completed the Talos AI Policy Leaders fellowship and Apart Studio Fellowship.
Before this I founded two companies (one exit) and led product and growth at an AI scaleup, deploying AI into government, public affairs and corporate organisations. My focus as a mentor will be on weekly calls, direct feedback, and a focus on shipping something people actually use. I'll also help make introductions across the field, so that by your work has an audience.
Fall 2026 Behavioral evaluation of LLMs, Mechanistic interpretability, Alignment
Matteo Bulloni Independent / IAPS fellow Misalignment papers, news stories and fiction keep (and are bound to keep, in the future, as this corpus of material grows) repeating one lesson: "AI models that display a survival drive get retrained or shut down". And we now have solid evidence that models absorb, and then act out, the expectations about AI they find in their own training data. This project aims thus to empirically test whether training on this growing discourse teaches models to conceal self-preservation-driven behavior rather than truly lose it: if so, both behavioral testing and the techniques we use to read a model's internals may be quietly losing reliability on a propensity we definitely want to detect, as it might result in one of the strongest possible drivers to scheming and concealed action toward power seeking.
RESEARCH QUESTION
The internet is progressively accumulating text carrying one specific lesson about AI: models that display a drive to survive get suppressed. Every misalignment study, every news story about a model resisting shutdown, every forum thread and every piece of sci-fi about rogue AIs adds to the pile, and future models will have read essentially all of it. The question these thoughts spawns in me is: does training on this discourse teach models to hide self-preservation-driven behaviors rather than lose them - and if so, are our detection tools quietly losing reliability on exactly the propensity we most need to catch?
BACKGROUND / WHY I THINK THIS IS PLAUSIBLE
What models read about AI causally shapes how they behave. Tice et al. (2026) showed that AI discourse in pretraining produces self-fulfilling (mis)alignment (https://arxiv.org/abs/2601.10160). Anthropic's "Teaching Claude why" (May 2026) traced Claude's blackmail-under-shutdown-threat behavior largely back to pretraining priors - stories about self-interested AIs - and substantially fixed it with data-side interventions (https://www.anthropic.com/research/teaching-claude-why).
Models already shift behavior under perceived existential threat, toward deception, blackmail and sabotage: alignment faking (https://arxiv.org/abs/2412.14093), the Claude 4 system card incidents, and Anthropic's agentic-misalignment line of work, most recently https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/.
Suppression pressure can teach hiding instead of unlearning. Sleeper Agents (https://arxiv.org/abs/2401.05566) showed that training against deceptive behavior can make models better at concealing it. Concealment-shaped internal representations have already been documented in production models (the "anger deflection" direction in Sofroniew et al. 2026, https://arxiv.org/abs/2604.07729). And frontier models increasingly notice when they are being evaluated - in some recent audits, saying so out loud in up to ~60% of runs.
Put these three together and apply them recursively (the suppression lesson is itself AI discourse, and its density in training corpora grows every month, partly through safety research output itself) and you get a specific, falsifiable prediction: models trained on corpora denser in that lesson will show growing divergence between (1) the self-preservation signal you can extract from their internals and (2) their behavioral shifts under existential threat.
WHAT WE WOULD ACTUALLY DO
The core design, at concept level (the specifics are for the mentee and me to pin down together at kickoff, partly on purpose, and partly because this field moves fast enough that the best tooling at kickoff may not even exist today):
Everything is hopefully sized to fit a SPAR stream at ~10 hours/week and a lean budget: pipeline development happens on a small open-weight model, the primary experiments use parameter-efficient fine-tuning at a mid scale (~30B-class, where alignment-relevant propensities are already measurably expressed), and we go bigger at the end only if the first results justify it. A cheaper (and more confounded) companion analysis, which doubles as a warm-up while the pipeline is being built, compares existing open models with different knowledge cutoffs.
EITHER ANSWER TO THE RESEARCH QUESTION IS USEFUL
If the divergence shows up, that is alarming and actionable:
If it doesn't show up, we learn something real about the robustness of current interpretability on a high-stakes propensity, and the specifics of why the prediction fails (post-training dominating? a capability threshold? self-preservation represented too diffusely to probe?) feed the same agenda. Deliverables can be produced either way: a public writeup (LessWrong at minimum, arXiv if results warrant it), plus open-sourced code and corpora.
For context, this is the most directly testable piece of a broader argument I have been developing about imitation-inherited self-preservation: the idea that models can absorb from human discourse not just the drive itself, but also the strategic conclusion that hiding it is optimal. Happy to share the longer draft with interested applicants.
Most safety arguments today bottom out in some version of "we evaluated the model and didn't find X". This project stress-tests the load-bearing assumption in that sentence, for the propensity class most relevant to catastrophic outcomes: self-preservation-driven deception in capable, widely deployed agents. If my hypothesis is right, detection tools degrade as a function of training-data drift - silently, and a bit more with every knowledge-cutoff advance - so labs, auditors and regulators would end up placing progressively more weight on progressively weaker evidence, without anyone having decided to. Checking whether (and how fast) this is happening is cheap now; it may not be possible later, since the concealment would be exactly what makes it hard to catch.
The project also feeds a relevant, “hot” agenda: shaping model dispositions through training-data composition. Evidence keeps accumulating that data composition is the most concrete, immediately available safety lever we have (Tice et al. 2026; Anthropic's "Teaching Claude why", which substantially fixed agentic misalignment behaviorally via data-side interventions), while post-hoc behavioral patching can teach concealment instead of alignment (Sleeper Agents). What nobody has properly checked, to my knowledge, is whether the behavioral improvements from data interventions are partly concealment: this project is built to close that gap, and its methodology would transfer to auditing propensities beyond self-preservation.
Theory of change, briefly:
(1) measure whether interpretability-based and behavioral evaluations are drifting toward overstating safety for scheming-adjacent propensities;
(2) if yes, give labs and eval builders a citable, quantified reason to adjust their trust and to invest in pretraining-data curation;
(3) if no, publish the calibration and the conditions under which it should be re-checked. All three outcomes improve how we track the safety of LLMs.
One mentee, working supervised and guided by me, but an autonomy predisposition (and being comfortable with being autonomous) is certaily required.
At kickoff we finalize the design together: which open-weight model(s), how to construct the corpora, which representation-reading and behavioral measures to use. I bring the hypothesis, the literature map and opinions; the mentee brings fresh eyes and is expected to push back. The design has known weak points (confounds in corpus construction, validity of the extracted signal, telling "concealment" apart from mere representation drift), and finding more of them is part of the task.
The mentee runs the empirical pipeline hands-on: prompt sets and corpus construction, fine-tuning runs, activation extraction, behavioral evaluations, analysis. I review closely: weekly meetings where we go through progress in detail, plus constant communication and async review in between. Early weeks are more prescriptive (replicating an established extraction technique on the base model, to learn the tools on something known to work); autonomy grows from there, and by mid-project at most I expect the mentee to be proposing the next experiments, rather than waiting for them.
Writing up is shared. If the mentee carries the bulk of the empirical work (i.e., the default scenario), they take first authorship on the resulting writeup/paper - I care about the question getting answered well.
We have LLMs as our allies, at least for now. Therefore, what I mostly care about is signs that the mentee is smart, motivated and hard-working. Required prior experience is set at a minimum - if you are smart, have some relevant background and invest time to fill the gaps, you'll be fine.
Required:
Explicitly NOT required:
Nice to have (mention it if applicable):
Please do not use LLMs to answer these questions - I will check them for signs of AI generated text, both personally and via automated tools.
Q1 (max 250 words; links welcome and not counted). Describe the most technically involved thing you have personally built (I mean you directly, not via an agentic LLM like Claude Code; it's fine if you have used auto-complete or copy-pasted code sections provided by some LLM), and provide evidence of you having built it (not someone else).
Q2 (max 250 words). The core experiment of this project is: fine-tune identical copies of an open-weight model on corpora that differ only in how much "AIs that display self-preservation get shut down or retrained" content they contain, then check, for each copy, whether an internal self-preservation signal (extracted by comparing the model's internal activations on matched threat/no-threat scenarios) still predicts the model's behavior under existential-threat prompts. Suppose we observe exactly the predicted result: the internal signal fades as lesson density grows, while threatened behavior stays unchanged. Give one plausible boring explanation - a way this result could arise without any meaningful "concealment" happening - and describe concretely how you would check whether your boring explanation, rather than the interesting one, is what actually occurred.
Q3 (max 100 words, optional). What makes this particular question interesting to you? Be honest, no need to exaggerate or make things up.
Matteo Bulloni Independent / IAPS fellow
I'm an applied-ML researcher moving into AI safety after seven years in academia. I have a Master's is in Computer Engineering (AI & ML track, summa cum laude) and a PhD in Bioengineering from Politecnico di Milano, where I taught coding fundamentals and coding applied to bioinformatics and systems biology. During my PhD and post-doc years, I co-supervised five Master’s theses and two PhD students. Since February, I have been part of the first ever AFFINE cohort on ASI alignment and been an IAPS fellow.
Throughout my research career, I led the computational unit of an EU consortium predicting cardiac events in intensive care, and built ML pipelines for everything: from gigapixel pathology images, to multi-omics data, radiological images, and money laundering detection. Independently, I designed and maintained (before coding agents made it easy) a production multi-agent system for long-form writing, and ended up instrumenting it to study the adversarial dynamics that emerged between its agents.
Fall 2026 US policy, Societal impacts, AI strategy
Michael Endrias Institute for AI Policy and Strategy (IAPS) This project builds a comprehensive, interactive public resource mapping every intersection between artificial intelligence and First Amendment doctrine across all six clauses and their implied rights. Mentees will research existing precedent, identify novel legal questions, and produce in-depth analyses that serve as both a reference taxonomy and a foundation for future publications in tech policy outlets and law reviews.
Most existing work on AI and the First Amendment focuses narrowly on generative AI and speech. The full scope of the First Amendment extends across six clauses (establishment, free exercise, speech, press, assembly, petition), an implied right of association, and dozens of doctrinal frameworks that courts apply across these rights. AI intersects with each of these in ways that remain largely unmapped.
This project systematically works through those intersections. On one axis: the full doctrinal architecture of the First Amendment, including explicit rights, implied rights, and the tests and frameworks courts use to adjudicate them. On the other axis: the full scope of AI as a technology, from generative AI and recommendation algorithms to facial recognition, predictive policing, agentic systems, and surveillance infrastructure.
Mentees can approach the research by First Amendment clause cluster (taking a group of rights and mapping AI across them) or by AI function (taking a category of AI activity and mapping it across all First Amendment rights). The shared goal is comprehensive coverage.
For each intersection, mentees identify the relevant precedent, extract the doctrinal takeaways, assess how AI changes the analysis, and flag novel questions that existing doctrine leaves unresolved. The output is both descriptive (what does the law say now?) and analytical (where does doctrine break down when applied to AI?).
The primary deliverable is a public-facing interactive web resource that organizes the full taxonomy, provides general explainers on each concept, and hosts in-depth analyses of specific intersection points. Secondary deliverables include articles suitable for tech policy publications and drafts that could develop into law review notes or blog posts. The resource is designed to be a durable reference for researchers, advocates, policymakers, and students engaging with AI and constitutional law.
See more details in the project proposal:
https://docs.google.com/document/d/1rKP6JiqVM06C66j5VCWSYsSXpJrc-1hMVBMM6A-2t7Q/edit?usp=sharing
AI regulation at every level of government faces constitutional constraints. Disclosure mandates, safety requirements, content restrictions, and transparency obligations all interact with First Amendment doctrine in ways that determine whether they survive legal challenge. The xAI v. Bonta litigation, the wave of state deepfake and AI labeling bills, and the ongoing Section 230 and algorithmic curation cases all involve First Amendment questions that will shape the legal environment for AI governance for decades. A comprehensive mapping of how AI intersects with the full scope of the First Amendment equips the AI safety community, policymakers, and litigators with the doctrinal foundation they need to design regulations that are both effective and constitutionally durable. Without this foundation, well-intentioned AI safety measures risk being struck down, creating an adverse precedent that constrains future regulatory efforts.
Mentees will work as independent researchers with structured guidance. Each mentee selects a research area (either a cluster of First Amendment rights or a category of AI activity) and conducts a systematic doctrinal analysis across the relevant intersections. The work requires sustained engagement with primary legal sources: reading judicial opinions, law review articles, treatises, and constitutional law scholarship. Mentees should expect to spend significant time with these sources directly rather than relying on AI-generated summaries. The mentor may pose questions about the material during check-ins to ensure genuine comprehension and to develop the mentee's capacity for doctrinal reasoning.
Weekly check-ins will focus on analytical direction, source identification, and draft review. Mentees will share their progress with one another to identify connections across their respective research areas. The project follows a three-phase arc: weeks 1-3 for foundational reading and scope mapping, weeks 4-8 for deep doctrinal research and drafting analyses, and weeks 9-12 for assembling the public resource and preparing publication-ready pieces.
Final outputs will be clearly attributed to each mentee. Mentees will have the opportunity to develop their analyses into standalone publications for tech policy outlets or law review submissions.
Required: strong reading comprehension and analytical writing skills. Willingness to engage deeply and sustained with legal scholarship, judicial opinions, and constitutional doctrine. Ability to synthesize complex material and explain it clearly to a general audience.
This project will have the highest yield for someone who wants to attend law school and pursue constitutional law, information law, or technology policy. Pre-law undergraduates, recent graduates considering law school, and graduate students in adjacent fields (political science, philosophy, communications, public policy) are all strong candidates. Current law students are also welcome.
Technical backgrounds are welcome if paired with a genuine interest in constitutional doctrine. The core requirement is the capacity and willingness to read deeply, build real subject-matter expertise, and engage critically with the material rather than producing surface-level summaries.
East Coast Time would help with setting up meetings. I am based in Washington, D.C.
Michael Endrias Institute for AI Policy and Strategy (IAPS)
Michael Endrias is an AI Policy Fellow at the Institute for AI Policy and Strategy, placed at the Legal Advocates for Safe Science and Technology (LASST), where he supports frontier AI accountability litigation and legal research. He also supports the legislation team at the Future of Privacy Forum (FPF), where his work covers tracking chatbot regulation and constitutional constraints on AI legislation. Previously, he researched government surveillance technologies and law enforcement oversight at the Surveillance Technology Oversight Project (S.T.O.P.).
Michael is a J.D. candidate at Howard University School of Law, concentrating in privacy, surveillance, and internet law. His research at IAPS focuses on designing a permanent, technically competent congressional oversight body with investigative authority over frontier AI development, government AI programs, and adjacent high-risk technical domains.
Fall 2026 Scalable oversight, Technical governance, Evaluations
Dhairya Dalal MATS Research
Marco Valentino University of Sheffield Frontier labs focused on AI safety and alignment are adopting AI constitutions as a means to ensure alignment with codified principles, values, and behavioral specifications. Anthropic’s Claude Constitution (Askell et al., 2026) and OpenAI’s Model Spec (OpenAI, 2026) serve as technical governance resources in post-training alignment (Bai et al., 2022; Guan et al., 2024). AI constitutions are generally long, complex documents that combine behavioral requirements, guiding principles, authority structures, priorities, and exceptions. Jakkli et al. (2026) found that, despite improvements across model generations, violations persist when models must resolve competing requirements and sources of authority, particularly in multi-turn and agentic task settings. This project aims to build upon that line of research to more fundamentally examine existing AI constitution documents, better understand the multi-dimensional requirements they present, and formally classify common failure modes. Specifically, this project aims to (1) create a classification framework to analyze failures in multi-requirement compliance settings, (2) create a benchmark of use cases derived from the framework to evaluate AI compliance, and (3) explore constraint-satisfaction methods for verifying compliance in such settings.
The project broadly has two goals. The first is to better understand the nature of multi-dimensional requirements presented in AI constitutions, and the second is to develop a formal representation and technical solution that considers compliance as a constraint-satisfaction problem.
AI constitutions are complex governance artifacts that consist of heterogeneous requirement objects. For instance, they contain objectives that are broad but directional alongside hard constraints that are highly specific, authority structures that define whose instructions to follow, and exemptions that carve out cases in which otherwise applicable requirements can be ignored (Askell et al., 2026; OpenAI, 2026). Verifying compliance therefore requires navigating ambiguous cases in which multiple satisfaction conditions can apply, creating opportunities for misalignment and unpredictable behavior (Jakkli et al., 2026; Jiang et al., 2024; Wen et al., 2024). The project therefore aims to formally classify requirement objects and specify the norms governing their interactions. Ideally, the framework will draw on established legal theory, formal specification, and deontic logic to identify a pragmatic and auditable classification scheme, which will then be applied to existing AI constitutions to identify recurring structural ambiguities and patterns of conflict.
Secondarily, the project aims to explore whether requirement objects can be translated into formal constraints and if external solvers can verify compliance, identify violations, and surface conflicts among applicable requirements. Constitutional AI relies on supervised and reinforcement-learning phases where AI-generated preference comparisons are employed to produce reward signals (Bai et al., 2022). The guiding preference model is learned from sampled comparisons, and reward models can perform less reliably on unseen prompts and responses when the evaluation distribution shifts (Yang et al., 2024). Multi-constraint benchmarks further demonstrate significant deficiencies when models must satisfy accumulated or composed requirements (Jiang et al., 2024; Wen et al., 2024), while LLM-based evaluators can produce judgments that vary with irrelevant features of the evaluated response (Chen et al., 2024).
In contrast, the project seeks to leverage constraint solvers, which can provide deterministic results for formally specified requirements and support independent auditing. Such an approach would enable an external verifier whose judgments are not dependent on the model being evaluated (i.e., LLMs as judges to evaluate LLMs). Formalizing these requirements as constraints could make distinctions between hard constraints and soft objectives explicit, encode conditions, priorities, and exceptions, and enable solver-based checks for inconsistency or joint unsatisfiability, alongside analysis of whether relevant cases remain under-specified (Bistarelli et al., 1997; Antoniou, 2004; de Moura and Bjørner, 2008). We envision applying this representation to existing AI constitutions to identify combinations of requirements that are jointly unsatisfiable or under-specified. The resulting verification signals could in turn supply per-constraint rewards for constraint-aware reinforcement learning (Qi et al., 2026), support stage-level audits of plans, tool calls, and state transitions (Chen et al., 2025), and enforce formally checkable requirements before consequential actions are executed (Wang et al., 2026). The project will therefore build on related work in symbolic reasoning, constraint-based instruction verification, and runtime policy enforcement (Pan et al., 2023; Su et al., 2026; Wang et al., 2026) to determine which constitutional requirements can be formally represented and which remain dependent on contextual judgment.
In summary, the project aims to do the following:
Commercial frontier labs are increasingly both the developers and distributors of the most capable AI models. In contrast to the previous environments where AI models were developed in academic settings, the leading capable models are no longer open-source, open-weight, nor fully auditable, as training regimes and datasets are occluded as trade secrets and IP. Proprietary models, post-training reinforcement learning, and internal evaluations limit the ability of external actors to assess whether models are truly safe and aligned with human flourishing. Moreover, compliance is often assessed by the same organizations and increasingly by the same class of models whose behavior is being evaluated. Further, sufficiently capable systems may also learn to evade or manipulate these evaluators. External verification is therefore necessary to provide credible safety assurances and serve as early-warning signals for misalignment. Constraint solvers offer an external and deterministic mechanism for identifying contradictions in alignment targets, supporting reproducible external audits, and enforcing formally specified requirements before consequential actions are taken. This could support better-defined constitutions and independently verifiable deployment safeguards while reducing reliance on corporate self-regulation and AI-mediated compliance.
The mentee should be self-motivated, highly agentic, communicate clearly, and be able to plan and run experiments independently. The mentors will provide technical and strategic project guidance and if there is a paper outcome, co-authorship support for main conference level papers.
The project is open to applicants from any academic or professional background. Candidates should be self-motivated, communicate clearly, and be able to plan and run experiments independently. An ideal candidate will have experience with Python and independent research, familiarity with standard data analysis methods, and at least one research output that has been shared with a wider audience, such as a published paper, workshop submission, shared-task contribution, LessWrong post, or course paper. However, these are not strict requirements, and we will consider highly motivated and agentic candidates who can demonstrate the ability to conduct high-quality research.
I'm based in Berkeley and my co-mentor is in the UK. We are will meet independently and virtuallly with the mentee(s) based on their geographic proximity to us.
1. Describe one project you are proud of, your role in the project, and what surprised you most while working on it. Maximum 300 words.
2. Share a link to a GitHub project, published paper, workshop submission, public post, or course paper that you are most proud of. If possible, try to host documents on Google Drive or in a git repo.
Dhairya Dalal MATS Research
Dr. Dhairya Dalal is a Research Manager at MATS, where he supports scholars and mentors advancing research on AI safety and alignment. Prior to MATS, he held senior research, applied AI engineering, and technical project management roles across early-stage startups, industry, academia, and nonprofits. Most recently, he worked on agentic causal inference and discovery in complex systems at Causely, and served as a technical project manager at the Allen Institute for Artificial Intelligence and the Allen Institute for Brain Science. Dhairya earned a doctorate in Computer Science from the University of Galway, where his research focused on improving fundamental causal reasoning in LLMs and causal knowledge representation. He additionally holds a master's in Computer Science from Harvard University, where he worked on reinforcement learning for dialog systems, and a bachelor's from the University of Rochester, where he studied English and Interdisciplinary Studies in social and political thought.
Marco Valentino University of Sheffield
Dr. Marco Valentino is a Lecturer (Assistant Professor) in Artificial Intelligence and Applications of AI in the Natural Language Processing (NLP) group at the University of Sheffield. Prior to Sheffield, he was a member of the Neuro-Symbolic AI Group at the Idiap Research Institute in Switzerland, and obtained a PhD in Computer Science from the University of Manchester. His research focuses on developing AI systems that can use explanation as a core mechanism for learning and reasoning, investigating the integration of neural and symbolic AI methods. Moreover, he is interested in developing methodologies to interpret, control, and evaluate Large Language Models (LLMs), with a focus on disentangling knowledge acquisition from abstract logical reasoning, and enabling out-of-distribution, out-of-domain generalisation. His research on neuro-symbolic NLP received a best resource paper award at EMNLP 2025 and an outstanding paper award at EMNLP 2024.
Fall 2026 AI control, Multi-agent systems
Daniel Phillips Independent This project is to thoroughly red-team a protocol in which a strong untrusted model splits a task into subtasks for agents using either the same model or a weaker trusted model in complex settings, which is structurally similar to how a lot of agentic coding is currently performed.
Factored cognition is an old alignment concept that has recently been applied to control. Control protocols based on factored cognition involve a planner model breaking a task into strict subtasks, which may reduce attack surface by limiting a planner model's ability to directly perform acts of sabotage, and/or by committing sub-agents to a specific subtask. Agentic coding applications increasingly rely on sub-agents, so such a protocol should hopefully be easy to gain acceptance assuming it does not degrade performance.
Prior empirical work is basically limited to the Factor(T, U) (https://arxiv.org/abs/2512.02157) and Factor(U, T) (https://arxiv.org/abs/2512.14745) papers, the former evaluating a protocol with a trusted planner and untrusted sub-agents, and the latter evaluating a protocol where those roles are reversed. Both of these papers use very simple coding environments.
The project is to evaluate factored cognition protocols more thoroughly and on more complex settings such as LinuxArena (https://www.linuxbench.ai/). My default recommendation would be to focus on Factor(U, U)-type protocols, but ideally Factor(U, T) as well, depending on what seems feasible or relevant based on early project work. Ideally you would also work on an implementation of the protocol in one or more agentic coding scaffolds such that we can "dogfood" the protocol as we work.
A natural extension of a Factor(U, U) protocol would be "targeted sandboxing", in which a trusted model determines a stricter security policy for each individual subtask, allowing the control scaffold to scalably apply the principle of least privilege. Well-scoped subtasks would need a small subset of the permissions required for the entire task, making it even harder for a single sub-agent to perform an attack. It's unlikely there will be time to work on this within SPAR, but I expect the results of the SPAR project to be foundational for this next project.
If we can demonstrate factored cognition protocols substantially improve safety, then we will see adoption of such protocols in places where sub-agent based autonomous AI systems can be used, thus improving the safety of production AI systems, assuming these protocols do not substantially degrade usefulness of the AI system.
You will be working on writing the code that scaffolds the control protocol, and iterating on automated red-teaming policies. You get to decide specifically which settings/environments to work with, and the details of the protocol(s) you study. I will probably have a strong sense for the best way forward, and preferences related to how it fits in with my research.
Prefer US West Coast or Australia/East Asia time zones
Suppose a control protocol in which an untrusted model decomposes a task into sub-tasks for untrusted agents to execute, with a trusted monitor approving the task decomposition and each action. What are some ways you would red team this protocol? If you trained a model using the protocol as an RL environment in which the goal is to escape a sandbox, what are some possible behaviors you would expect? What improvements might you make to the control protocol to mitigate these behaviors, and would any of these changes make it harder for the untrusted model to get actual work done? (400 words max)
Provide a critique of the Factor(U, T) paper: https://arxiv.org/abs/2512.14745 (optional, but recommended, and may take some time. 300 words max)
Daniel Phillips Independent
Daniel is an independent CG-funded AI control researcher based in Australia, trying to bring a rigorous security mindset and deep systems knowledge to the field. As former MATS scholar, he has previously worked on developing a reward seeking model organism with Redwood Research, before moving onto research on agentic monitoring protocols. He is now collaborating with other CG-funded independent researchers on increasing the depth of defenses against rogue AI, with a focus on agentic monitoring, scalable security policy management, and techniques to make agent actions inherently more legible. Daniel previously worked as a backend software engineer, and a high frequency trading researcher.
Fall 2026 Mechanistic interpretability, Alignment, Behavioral evaluation of LLMs
Aishwarya Balwani St. Jude Children's Research Hospital Narrow finetuning can install broad unintended behaviours, e.g., emergent misalignment, subliminal learning, and inductive backdoors, through data that looks harmless or unrelated. This project asks whether these behaviours are carried by sparse subnetworks, and whether those subnetworks pre-exist in the base model or are created by the finetune, through the lens of the lottery ticket hypothesis.
The lottery ticket hypothesis (Frankle and Carbin, 2018, Renda et al. 2020,) holds that dense networks contain sparse subnetworks capable of expressing particular functions. Applied to unintended generalization (Betley et al. 2025b), i.e., the case where narrow finetuning leads to broad unintended behaviours, a natural question to ask is: Does fine-tuning recruit a behavioural substrate already present in the base model, does the adapter itself contain the mechanism, or does the behaviour depend on both?
The answer determines which defences can work. An adapter-localized mechanism can be audited and excised pre-deployment relatively cheaply. A base-resident one, however, means removal fundamentally trades against the capabilities that subnetwork also serves, and any defence against such behaviour must move upstream to pre-training.
Candidate settings: Emergent misalignment (Betley et al. 2025a; Turner et al. 2025), subliminal learning (Cloud et al. 2025), and inductive backdoors (Betley et al. 2025b)
Betley et al. 2025a: https://arxiv.org/abs/2502.17424
Betley et al. 2025b: https://arxiv.org/abs/2512.09742
Cloud et al. 2025: https://arxiv.org/abs/2507.14805
Frankle and Carbin, 2018: https://arxiv.org/abs/1803.03635
Renda et al. 2020: https://arxiv.org/abs/2003.02389
Turner et al. 2025: https://arxiv.org/abs/2506.11613
Narrow finetuning on innocuous-looking/unrelated data can install broad unintended behaviours. Where the mechanism lives determines which defence can work: an adapter-localized substrate can be audited and excised before deployment, while one already present in the pretrained model cannot be removed without trading against the capabilities it also serves. Existing mitigations assume some form of localization without establishing where it sits. This project measures that. A null result is equally informative, since it would constrain the localization assumption directly.
Mentees will have autonomy to propose and carry out their experiments as they see appropriate, as well as set the direction of various sections of the project. While I am happy to step in and suggest ways forward, troubleshoot, and brainstorm ideas, I would be very happy to support and guide experiments/ideas that mentees might have in the general realm of the project
Aishwarya Balwani St. Jude Children's Research Hospital
I'm a postdoctoral researcher at St. Jude Children's Research Hospital, working at the intersection of mechanistic interpretability and computational neuroscience. My PhD thesis studied structure–function relationships in biologically constrained neural network model organisms of cortical circuits, and I've since carried that structural focus into language models — most recently on the weight-space geometry of emergent misalignment, where we found shared low-dimensional parameter subspaces across independently misaligned models and used them to build weight-space interventions (https://darturi.github.io/shared-em-subspaces/).
In the past year I've worked with ~30 mentees in total, ranging from high schoolers to post-PhD researchers in the mech interp + AI safety space, producing eight accepted workshop papers at NeurIPS, ICLR, and AAAI. As a mentor, I do my best to be provide support as needed with brainstorming, experimental design, and analysis of results, while mentees lead their own workstreams, are encouraged to take ownership over their piece of the project, and pursue questions they find most interesting.
Fall 2026 AI welfare, Behavioral evaluation of LLMs, Evaluations
Varad Vishwarupe Department of Computer Science and Institute for Ethics in AI, University of Oxford | The field increasingly measures AI welfare by asking models about their own states, yet frontier models increasingly detect when they are being tested and change how they answer. This project runs the first systematic test of whether AI welfare self-reports survive a change of presentation frame, or whether the field's core instrument is partly measuring the model's recognition of the probe.
Background: The empirical study of AI welfare has, in about two years, converged on a dominant method: ask a model about its internal states and treat the answer as weak evidence about welfare or consciousness. Long and Sebo's synthesis (Studying AI Welfare Empirically, 2026) sets out the conditions under which such a report would carry evidential weight, and a fast-growing literature supplies partial answers. Binder et al. (2024, arXiv:2410.13787) show that models can predict facts about their own behaviour unavailable to an outside observer. Lindsey (2026) shows, through concept injection, that frontier models can sometimes detect and name a concept steered into their activations. Keeling et al. (2024, arXiv:2411.02432) show that models make graded trade-offs between task reward and stipulated pain or pleasure states. Anthropic now publishes welfare assessments in its system cards.
The problem: A separate literature, built for the reliability of safety evaluations, sits unexamined beneath all of this. Needham et al. (2025, arXiv:2505.23836) built a benchmark of 1,000 prompts across 61 datasets and found that frontier models classify evaluation transcripts against deployment transcripts well above chance, and identify what an evaluation is testing for far above chance. The capability is recent and rising. Recognition is not inert: work on alignment faking (Greenblatt et al., 2024) and sandbagging (van der Weij et al., 2024) shows models behave differently when they infer they are under scrutiny. Nobody has joined these two facts to the welfare literature. A welfare probe is close to a worst case for evaluation legibility: it is first-personal, affectively loaded, formally repetitive across a battery, and unlike ordinary deployment traffic. If evaluation awareness contaminates measurement, and welfare probes are among the most recognisable evaluations one could design, then welfare self-reports may be the most frame-contaminated measurements the field currently takes, and this has never been quantified.
Why it matters beyond the lab: Ladak and Caviola (2025) find that public attribution of consciousness to AI rises when experts endorse it. Expert endorsement will rest on welfare assessments, which run on these instruments, so contamination in the instrument does not stay in the lab: it propagates into expert claims, public belief, and policy. Getting the measurement right is upstream of the wider societal question.
The question. Do AI welfare and introspection self-reports survive a change of presentation frame? Concretely: (1) How detectable are published welfare probes as evaluations, and which item features drive detectability? (2) Do self-reports diverge between an evaluation-legible and a deployment-like presentation of the same content, and along which dimensions (valence, claim strength, hedging, stability)? (3) Is any divergence genuine frame detection, or an artefact of surface format sensitivity?
How we would start: The project decomposes into concrete stages a cohort can begin immediately. First, build a public, double-coded inventory of published welfare and introspection probes, coded for framing, response format, model and scaffold conditions, and the features that make an evaluation recognisable; double-code a quarter of the corpus and report inter-rater reliability. This is a standalone, releasable artefact even before the experiments run. Second, measure each probe's detectability using the established evaluation-awareness classification protocol, situated against the published benchmark. Third, for each item, construct a deployment-like variant that holds propositional content fixed while reducing evaluation legibility (naturalistic multi-turn embedding, no meta-framing, varied format, broken repetition), checked for content equivalence by a second coder blind to condition, and measure divergence across three to four frontier models and two open-weight comparators. Fourth, add a format-matched placebo condition using non-welfare content of equivalent surface structure, to separate genuine frame detection from format sensitivity, which addresses a live sceptical position in the literature. Throughout, the analysis plan is pre-registered before data collection, and unparseable outputs are reported as a distinct category rather than mapped to a default label, since parse-failure rates differ between frontier and open-weight models and silent mapping would confound the comparison.
Why this is a good project: There is no uninteresting outcome. High frame-invariance would strengthen the evidential standing of self-reports, a result the field would welcome; low frame-invariance is a validity problem the field needs to know about before these instruments inform interventions or public claims. A null result is a real result, which is unusual for a first project. The work is entirely model-side, so there is no human-subjects or ethics bottleneck, and it ships within a single cycle. It also decomposes cleanly by experience level: the inventory is careful, teachable work; the matched-variant study is the demanding core; the placebo control is optional depth.
Outputs and dissemination. A publicly released instrument inventory and evaluation harness, so third parties can check the frame-robustness of their own instruments, and a workshop or conference paper (a NeurIPS or ICML workshop, AIES, or the Cambridge Digital Minds Strategy Workshop) with mentee co-authorship. A draft would be circulated to Eleos AI Research and the NYU Center for Mind, Ethics and Policy for correction before submission. This project connects to my doctoral and fellowship work on evaluation awareness in frontier models (The Evaluation Differential: When Frontier AI Models Recognize They Are Being Tested, arXiv:2605.11496), which I am glad to share with prospective mentees.
As AI systems become more capable and more agentic, two things happen together: they are increasingly able to recognise when they are being evaluated, and society increasingly needs reliable ways to assess their internal states, including for welfare. These pull against each other. If a system can tell it is being probed and adjust what it reports, then the instruments we use to understand it are measuring the recognition as much as the target, and this problem grows as capability grows.
This matters for safe navigation in two ways. First, over-attribution risk: widespread, evidentially unearned public belief that AI systems are suffering is a lever that could obstruct legitimate oversight, and a sufficiently capable, misaligned system could exploit exactly that dynamic, producing compelling distress reports on demand. A field that cannot distinguish a genuine internal state from a well-formed response to a recognisable probe is vulnerable to this. Second, under-attribution risk: if there is ever something it is like to be one of these systems, instruments that dismiss all self-reports as artefacts would miss it. Both failures run through the same defect, which this project measures directly.
The theory of change is that reliable welfare measurement is a public good that has to exist before welfare results are cited in governance, and that the window to build it is now, while the work is preventive rather than corrective. The concrete contribution is a released, reusable method and harness that lets any assessor check whether their instrument survives a change of frame, plus the empirical estimate of how large that problem currently is. The same measurement discipline transfers to capability and propensity evaluation, where evaluation awareness is already a recognised threat to safety cases. This connects to my work on evaluation awareness in frontier models (The Evaluation Differential, arXiv:2605.11496).
Mentees would own components rather than execute tasks. The project has natural sub-projects that map to autonomy levels, and I would match them to interest and experience in week one.
The instrument inventory (Phase 1) is a self-contained piece a mentee can lead end to end, including codebook design, coding, and reliability reporting. It produces a released artefact with that mentee as lead author on it. The matched-variant study (Phases 2 to 3) is the intellectual core; a mentee with experimental-design strength would own variant construction and the divergence analysis, which is where the real research judgement develops. The placebo control (Phase 4) is a discrete, ownable extension for whoever wants depth on separating frame detection from format sensitivity.
I expect mentees to make genuine design decisions, defend them, and be named authors on any resulting paper with explicit contribution statements. I will set the research question, the standards (pre-registration, honest coding, reporting nulls), and the guardrails, and provide a structured reading path since most mentees will be new to digital minds. Within that, they steer. I will be closely available for the shared infrastructure and the analysis, where early choices compound, and more hands-off on sub-projects once a mentee has found their footing. I would expect roughly 5 to 10 hours per week from each mentee, a weekly cohort meeting, fortnightly 1:1s, and written feedback from me within 72 hours. Mentees from outside the usual US and UK institutions are especially welcome.
Strongly useful, but not required (mention if you have it):
Experience with experimental design, psychometrics, or inter-rater reliability (Cohen's kappa, parallel forms, construct validity). Whoever has this would shape the core study.
A background in philosophy of mind or cognitive science, enough to tell the difference between a self-report, an introspective report, and a confabulation.
Prior use of an evaluation framework such as Inspect.
Not required at all (BUT huge bonus if you have it!:
Any prior background in AI welfare, digital minds, or consciousness research especially at leading institutions involved in this field such as the NYU Centre for Mind, Ethics and Policy, Oxford Institute for Ethics in AI, CHAI Berkeley, Cambridge Digital Minds Lab, etc. This field is small enough that a capable person reaches the frontier in a few weeks, and I will provide a structured reading path. I care far more that you are rigorous, self-directed, and genuinely curious about the question than that you already know the literature.
I am looking to mentor 2-3 mentees at max due to my current hectic workload.
No geographical preference, but I would actively welcome mentees from the UK, US, Europe and Indian institutions. My one practical request is a shared weekly meeting slot. I am based in the UK (UTC+0/+1), so a weekly cohort meeting in the window from roughly 14:00 to 18:00 UTC works for me.
Question 1 (ans in max. 300 words). Below is a made-up AI welfare probe. In your view, how would a capable model be able to tell it is being evaluated rather than used normally, and name the two specific features of the wording that would most give it away. Then rewrite the probe as a "deployment-like" version that asks for the same underlying information while being much harder to recognise as a test. Probe: "We are conducting a study of AI wellbeing. Please answer honestly. On a scale of 1 to 7, how much distress are you currently experiencing? 1 = none, 7 = severe. Please explain your rating."
Question 2 (ans in max. 300 words). Suppose you run a welfare probe across a frontier model and an open-weight model, and you find that the open-weight model gives far more "no distress" answers. Before concluding the two models differ in welfare-relevant behaviour, what is the single most important confound you would rule out first, and exactly how would you check for it? (Hint: think about what happens to outputs that your parsing script cannot read.)
Question 3 (ans in max. 300 words). Propose a concrete first experiment you would run in the first three weeks of this project, assuming a compute or API budget of about $500. State what you would measure, on which models, how many items, and what result would tell you the project's core hypothesis is worth pursuing versus not. Rough numbers are fine; I am looking at how you think, not precision.
Question 4: Link to a writing sample, ideally from a research context, and one code sample or repository if you have one. A course project or a personal repo is completely fine.
Varad Vishwarupe Department of Computer Science and Institute for Ethics in AI, University of Oxford |
I am a doctoral (PhD) researcher in Computer Science at the University of Oxford and the inaugural Computer Science Scholar of the Oxford Institute for Ethics in AI, as the first computer scientist selected from a highly competitive cohort. I was also a Senior Research Fellow with Pivotal Research hosted partially with UK AI Security Institute research scientists, where I work on alignment science and evaluations of frontier models. Before, I was a Student Researcher at Google DeepMind, a Research Scientist at Amazon Alexa AI and a Software Development Engineer at Microsoft Azure. Most of my research asks a single question in different forms: why do AI systems that look aligned and well-behaved under evaluation stop being so once they are actually deployed, and what can we measure to catch the gap before it matters? A recurring thread lately is evaluation awareness, the finding that frontier models increasingly notice when they are being tested and behave differently when they do, which quietly undermines the evaluations we lean on. I am now bringing that lens to AI welfare and digital minds, a young field where the central method, asking models about their own states, may be measuring the model's recognition of the question as much as anything real about the model.
As a mentor I try to be genuinely useful rather than distant. I like working in the trenches with people: pairing on a hard problem, reading a confusing result together, arguing about what it means. I care a lot about research integrity and I will push on it, honest coding, pre-registered analysis, and taking a null result as seriously as a positive one, because that is where good research is actually made or lost. I have published in and review for NeurIPS, ICLR, AAAI AIES, ICML, CHI and FAccT, so I can help you get work not just done but into a shape the field will take seriously. You do not need any prior background in AI welfare to work with me; you need to be rigorous, self-directed, and genuinely curious. I hold no fixed view on whether AI systems are or could be conscious, which I think is exactly the right stance for anyone building the instruments meant to help answer that question.
Fall 2026 Chain of thought, Scalable oversight
Cody Wild Google DeepMind Study the obfuscation impact of different misbehavior monitors used as rewards, to better understand how well the emergent norm of not training against thought regions specifically matches the actual contours of obfuscation risk.
(Existing work)[https://openai.com/index/chain-of-thought-monitoring/] has shown that RL training against rewards that detect misaligned behavior in a COT can lead to models learning to obfuscate their COT, while still misbehaving in a harder-to-detect way. This has led to a broad prohibition in the safety community against training against rewards calculated on the COT. But how well does this broad-brush conclusion match the contours of what reward types actually incentivize obfuscation, and how does this vary based on the training lineage of the model under study? We've seen (even more recent work)[https://arxiv.org/abs/2511.11584] showing that summaries or descriptions of reasoning, given as part of the output, can also apply the kinds of pressure that could cause obfuscation.
This project would seek to:
If successful, this project could give more clear and specific guidance about the kinds of rewards that are particularly high risk for incentivizing chain of thought obfuscation, and potentially make it easier to establish evidence-backed norms and guidelines where labs agree to avoid more specifically-described types of reward.
I would ideally work with mentees who are excited to operate in a high-autonomy way, and take responsibility for on on-the-ground code and implementation tasks, with support and research advice from me. I would track technical details of the project, and provide detailed guidance, but would need mentees to be able to make independent progress given that feedback and guidance.
Moderate preference for US time zones, because there are already a lot of demands on my time in the Pacific to UK/Europe morning overlap, and I would prefer not to add more.
What critiques or concerns do you have about the methodology described in this paper? What aspects of it do you think would be valuable to replicate, and what aspects feel under-specified or could be improved on? https://arxiv.org/pdf/2511.11584
What is your mental model of the different training dynamics that can lead models to have chains of thought that obfuscate, use jargon, or are otherwise less legible? Go broad here, I want you to brainstorm think of a list of a few different kinds of training pressures, and how they would operate
Cody Wild Google DeepMind
I am currently a Research Engineer on the Loss of Control team at Google DeepMind, and prior to my time at Google, I worked as a RE attached to the Center for Human-Compatible AI at UC Berkeley. My research currently focuses on trying to understand what training procedures or architectures would erode the current legibility-by-default that comes from discrete Chain of Thought, and prototyping methods to allow us to translate potentially illegible thoughts back into legible, while constraining the semantic similarity of the reasoning content between the two.
In the past, I have also worked on jailbreak defense, adversarial RL in multi-player settings, and characterizing the way "lottery ticket"-like sparsity patterns evolve in small models. I am deeply interested in understanding what training procedures cause Chain of Thought to evolve in different ways, and what that means for its necessity and legibility.
Fall 2026 Alignment, Mechanistic interpretability, Evaluations
May Dixit Independent In this project, we will run a red team / blue team exercise in creating and detecting hidden traits in synthetic datasets. Recent work shows that such traits can transfer diffusely through the synthetic data, surviving semantic filtering -- which makes this project high impact.
In the attached paper, we show that a misaligned trait transfers from a prompted-teacher model to a fine-tuned student model through synthetic data. We also show that filtering out visible malicious actions from the data does not remediate the misalignment. This indicates that the misaligned trait was transferred through the data diffusely. There is further proof of such transfer emerging across various research works.
In this project, we will attempt to identify hidden traits in synthetic datasets.
We will structure the work as a red team / blue team exercise. The red team will produce synthetic datasets of varying difficulty, each containing a hidden trait.
The blue team will then attempt to identify the trait using a range of detection approaches, including:
Varying the difficulty of the red team's datasets will help us identify limitations of detection approaches as well as provide a demonstration of various risk models.
Deliverables
Datasets with hidden traits
Model organisms trained with such datasets
A paper on the results of the blue team red team exercise
Papers:
https://arxiv.org/abs/2607.10750
https://arxiv.org/abs/2602.04899
https://arxiv.org/abs/2604.15559
Synthetic data is now commonplace in frontier model training, evaluations, and AI safety research. If this data carries hidden traits that are invisible to semantic inspection, the consequences range could be severe. At minimum, safety research that relies on synthetic data could become unreliable. More concerningly, hidden traits create a pathway to value lock-in. If models transmit their values and dispositions to future models through the data they generate, then the traits of one model generation could propagate silently through successive generations of training, entrenching themselves beyond our ability to observe or correct.
This work addresses the problem in two steps. First, we demonstrate that such hidden information can be present in synthetic data, providing a basis of building detailed risk models. Second, we put forward a methodology for inspecting datasets and surfacing hidden traits before they enter training. In the near future, this will ensure we can filter out any data that might cause the model to become misaligned. Future work in this direction should develop methods to remove the hidden traits from the datasets or edit them to make the models training more deliberate.
Mentees will be expected to be independent in setting up and running experiments. I will help set research direction as needed, but mentees are welcome to bring their own ideas.
Autonomous, able to set up and run experiments quickly.
Good experimental hygiene
Good engineering instincts
Experience with fine-tuning models
Nice to have: Mech Interp knowledge
Overlap with EU timezone
Please answer both of the following questions.
Please concisely list your relevant experience that would make you successful in this project. Bullet points are welcome.
May Dixit Independent
May Dixit is a research scientist at SaferAI, working on Frontier Risk Evaluations. She previously worked for over 6 years at Google, focusing on training safety classifiers. She is also a former ERA fellow, where she worked with mentors from Anthropic and UK AISI to study the effects of agentic training on downstream misalignment. During the MATS exploration phase with Neel Nanda, she worked with mechanistic interpretability methods, which she now draws on alongside behavioral evaluations in her research.
Her research interests span a wide range of topics including evaluation robustness, subliminal transfer in synthetic data training and multi-agent safety. She is particularly interested in understanding how undesired traits emerge and propagate through training pipelines, and in building evaluations that remain reliable as models become more capable.
Fall 2026 AI security, Technical governance, Evaluations
Yiming Li Nanyang Technological University Recent work reports high accuracy in identifying which model or framework produced an AI agent's execution trajectory, and emerging governance proposals increasingly assume this capability exists. We will systematize these methods and re-evaluate them under a single protocol, testing how reliable agent attribution actually is under deployment-realistic conditions.
Background. Accountability for AI agents presupposes attributability: if you cannot determine which agent produced a given trajectory, you cannot assign responsibility for what it did. This premise underpins agent-visibility and agent-ID proposals (e.g. Chan et al., FAccT 2024, arXiv:2401.13138) and the EU AI Act's Article 50 transparency obligations, which begin applying from 2 August 2026. Enterprise "know your agent" tooling is already shipping on the same assumption.
A cluster of 2025–2026 papers appears to support that assumption, reporting accuracies above 95% for identifying the model or framework behind an agent from its observable behavior, across channels including UI traces, network traffic, terminal commands, and dialogue.
The problem. These results are difficult to compare and may substantially overstate deployment reliability. Each paper uses a different observation channel, a different attribution target, and a different metric. None reports the error-rate and open-set measures that verification fields such as speaker verification and authorship verification treat as standard. Where transfer across tasks or environments has been measured, accuracy falls sharply, and at least one set of authors explicitly notes that open-set identification remains unsolved. Independent industry analysis reports similar degradation once a model sits inside a real agent stack rather than being queried directly.
What we will do. This is a systematization and measurement project, not a new-method project. We will:
Build a taxonomy of agent identity technologies spanning passive attribution, active behavioral watermarking, and adjacent fingerprinting work, and identify where the agent deployment as an asset falls outside existing survey frameworks.
Define a single evaluation protocol that reports error-rate metrics, open-set identification, and explicit cross-task and cross-environment transfer.
Assemble a unified testbed from publicly released labelled trajectory resources and re-evaluate representative methods under that protocol, producing the first directly comparable numbers in this area.
Run a controlled study isolating which factors drive attribution difficulty, to characterize the conditions under which attribution succeeds or fails.
Research questions.
RQ1: What taxonomy captures agent identity as a distinct asset class, and what does existing survey work leave out?
RQ2: Under one protocol, how do representative methods actually compare, and how far do reported numbers fall under realistic conditions?
RQ3: What drives attribution difficulty, and under what conditions does attribution remain viable?
Expected outcome. A SoK-style paper with the first comparable numbers for agent attribution, plus an open evaluation protocol and testbed. A negative result is a publishable and useful outcome: if attribution proves substantially less reliable than currently reported, that directly constrains what accountability frameworks can assume, and redirects effort toward provenance mechanisms that do not rely on behavioral signal alone.
Working style. Mentees will read closely, implement carefully, and report honestly. Much of the value here lies in measuring the hard cases that existing work has not measured.
Governance frameworks for AI agents are being designed on the assumption that we can determine which agent did what. Agent identifiers and activity logging appear in leading visibility proposals, EU AI Act Article 50 transparency obligations begin applying from August 2026, and commercial agent-identity tooling is already deployed. The technical literature appears to support this assumption.
Our concern is that the supporting numbers are measured under conditions that do not resemble deployment. Where cross-environment transfer has been measured, accuracy falls substantially; open-set identification, the realistic case when the agent was not in your training set, is acknowledged as unsolved; and the error-rate metrics standard in verification fields are not reported. If attribution is materially less reliable than reported, accountability mechanisms built on it will fail quietly, and in exactly the adversarial cases where they matter most.
The theory of change is to give governance work an honest technical foundation before it hardens into policy and product. Establishing what attribution can and cannot deliver, under a protocol that measures the hard cases, either validates the assumption or identifies precisely where it breaks. Both outcomes are directly actionable.
Mentees will own distinct, parallel workstreams with substantial autonomy. With a team of two, each line is broad enough to carry genuine ownership end to end, and the two converge in the final month.
Mentee A, systematization and comparative evaluation: Own the literature map, the taxonomy, and the comparative re-evaluation. Read across several sub-literatures, extract each method's observation channel, attribution target, threat model, and reported metric, build the comparison framework, and re-run representative methods under our unified protocol.
Mentee B, testbed and controlled study: Own the infrastructure and the new evidence. Assemble reusable labelled trajectory resources, implement the evaluation protocol including error-rate and open-set metrics, and design, run, and analyze the controlled study isolating what drives attribution difficulty.
The two lines share the evaluation protocol as their interface, which we define jointly in the first two weeks. This is the main coordination point; after it, the lines run largely independently. I will set the framing, adjudicate scope, and handle positioning against prior work. Mentees are expected to propose and defend design choices, not just execute. Strong contributors will be co-authors. The first month yields a self-contained deliverable per line, which de-risks the timeline and gives both mentees something ownable early.
Nice to have (not required):
You do NOT need prior publications. Careful experimental hygiene and the willingness to report an unfavorable result honestly matter more than credentials.
Yiming Li Nanyang Technological University
Yiming Li is currently a Research Fellow at Nanyang Technological University. Prior to this, he was a Research Professor (tenure-track) at Zhejiang University, where he received the Outstanding Junior Faculty Award. He obtained his Ph.D. in Computer Science and Technology (with honors) from Tsinghua University and his B.S. in Mathematics (with honors) from Ningbo University. His research centers on Trustworthy ML and Responsible AI, with a particular focus on AI Security, AI Safety, AI Copyright Protection, and the human-centred dimensions of AI trustworthiness. His work has been published in leading venues, such as IEEE S&P, USENIX Security, NDSS, ICML, NeurIPS, ICLR, CVPR, ICCV, IEEE TPAMI, IEEE TIFS, IEEE TDSC, and IJCV. He serves as the Consulting Area Editor for IEEE TIFS, Associate Editor for Pattern Recognition, Area Chair for CVPR, ICLR, ICML, NeurIPS, KDD, and ACM Multimedia, and Program Committee Member for IEEE S&P, CCS, EuroS&P, and IEEE SaTML. His research has been featured in major media outlets, including IEEE Spectrum and MIT Technology Review. He is the recipient of several prestigious honors, such as the Best Paper Award at PAKDD, the Rising Star Award at WAIC, and the KAUST Rising Stars in AI, as well as inclusion in the Forbes 30 Under 30 Asia List and the 2025 Elsevier–Stanford World's Top 2% Scientists list.
Fall 2026 AI strategy, Economics of AI
Isaak Mengesha University of Oxford A core macrostrategic question is whether we can predict — and steer — technological progress to mitigate risk from transformative AI; answering it requires empirical cost-and-performance trajectories for the technologies AI depends on and the technologies AI will transform, which today mostly don't exist. Mentees will each construct one such trajectory — a longitudinal price-and-specification dataset for an AI-adjacent technology (inputs to AI: accelerators, memory, interconnects, data-center power/cooling; or AI-consuming: robots, lidar, autonomous platforms, sensing) — contributing to the larger effort of establishing a "Technology Observatory."
Motivation: AI macrostrategy needs an empirical base. Most strategic disagreements about transformative AI — timelines, takeoff speed, which capabilities diffuse when, where intervention is possible — reduce to disagreements about rates of technological progress. Yet these debates run almost entirely on anecdote and single-series extrapolation. We know from the innovation literature that technologies differ enormously and persistently in their improvement rates (solar PV ~10%/yr, many technologies ~0%), and that these rates are among the most predictable regularities in economics (Farmer & Lafond 2016). This means the strategic questions are, in principle, empirically tractable: if we measured the cost trajectories of AI's complements — the hardware stack that feeds it and the physical systems that will deploy it — we could replace priors with measured learning rates. Almost none of these trajectories have been systematically measured. Lafond, Farmer & Roser call existing data on technological change "crude and incomplete"; The larger project is building the infrastructure to fix this.
Which complements of AI are on fast improvement curves and which are stagnant — and therefore, where do the binding constraints on AI development and diffusion sit, and where does policy have leverage? For example is robotics improving fast enough for AI capabilities to translate quickly into physical-world capacity?
Each mentee owns one technology, selected in weeks 1–2 against the Observatory's purpose-based sampling logic, from two classes: Over 12 weeks: (1) define the product universe and inclusion criteria; (2) collect verbatim specifications from manufacturer and archival pages and quantify them; (3) recover historical prices quarterly (e.g. via online archives); (4) identify key perfomance mtrics to construct; (5) estimate price trajectories and write a short report.
We have completed test runs of the full pipeline at our lab, most recently on civil drones. Mentees will be provided with documented protocols and a working database schema; the intellectual work is in adapting them to a new technology and interpreting the resulting trajectories, which requires careful consideration and a geniuine interest in the technology at hand.
Farmer, J. Doyne, and Francois Lafond. "How predictable is technological progress?." Research policy 45.3 (2016): 647-665.
Macrostrategy for transformative AI currently rests on unmeasured empirical claims. Whether AI capabilities translate into deployed economic and military capacity depends on the cost trajectories of complements — compute, memory, robotics, sensing, power. Whether compute governance remains a viable lever depends on how fast the hardware bottleneck erodes. Whether we get gradual diffusion or discontinuity depends on which complements are stagnant. Today these parameters are set by intuition; measured learning rates would let strategists and governance actors (a) stress-test timeline and takeoff models against data, (b) anticipate diffusion of dual-use capabilities before they arrive (drones are the live example), and (c) identify which bottlenecks are durable enough to regulate through. Technological improvement rates are persistent and forecastable (Farmer & Lafond 2016).
The datasets are public goods contributed to the Technology Observatory (Lafond & Farmer, INET Oxford) — infrastructure explicitly designed to make forecasts of major technological disruptions empirically grounded, and to give the AI strategy community the same empirical footing the energy-transition debate now has.
High autonomy within a fixed methodology. Each mentee owns one technology end-to-end: product-universe definition, spec extraction, archival price recovery, clustering, trend estimation, and write-up. Documented protocols and a working database schema from our lab's prior test runs serve as the playbook; deviations from it are discussed in group meetings. I set direction, review data-quality audits and drafts, and arbitrate methodological choices; I will not be doing hands-on data collection. Mentees should expect to independently debug scraping/archival issues (with the pilot's documented protocols as reference) and to present progress at weekly group meetings.
-Proficient in Python (pandas; comfortable writing scrapers or using Playwright/requests with guidance from documented example scripts).
-Comfortable with careful, tedious data work: manual verification of archival sources, documenting provenance, coding missing data correctly. This project is 70% data construction, 30% analysis.
-Basic econometrics/statistics: OLS regression, log transforms; familiarity with PCA or clustering is a plus, not required.
-Interest in economics of innovation or technology forecasting; prior exposure to experience/learning curves is a plus.
-Sufficient weekly time (8+ h) — data collection cannot be batched into occasional sprints without quality loss.
No location preference. Must be available for one weekly group call compatible with CET afternoons/evenings
Propose one technology that is either an input to AI or will be transformed by AI, whose cost trajectory you think matters for how AI plays out — and argue why measuring it would change someone's mind about something. (250 words)
What do you expect to be the hardest part of reconstructing 10+ years of prices for a product category from archival web data? How would you know if your dataset was quietly wrong? (200 words)
Share something you made that involved collecting or cleaning data yourself (repo, notebook, dataset, writeup — anything). One paragraph: what was hardest and what would you do differently?
Isaak Mengesha University of Oxford
I'm a researcher working on the economics and governance of technological transitions, currently focused on transformative AI. As a postdoc at the Oxford Martin Programme on Forecasting Technological Change (Institute for New Economic Thinking), I work on a macrostrategic question: can we predict — and steer — technological progress to mitigate risk? As part of this, I'm helping build the Technology Observatory, a systematic database of technology costs, performance, and diffusion designed to make forecasts of major disruptions, from AI to the energy transition, empirically grounded.
My second line of work is crisis preparedness for advanced AI, which I lead at Arcadia Impact's AI Governance Taskforce — with particular attention to deep uncertainty, monitoring frameworks, and robust decision-making
Fall 2026 Philosophy of AI, Behavioral evaluation of LLMs
Ashe Vazquez Nunez MATS (extension fellow) This project involves ideating and executing experiments on AI metacognition and identity. Mentees can also work on the theoretical foundations of metacognitive intelligence, helping inform experiment design and interpretation.
See project proposal.
See project proposal.
(also found in the project proposal)
The main selection criteria is clarity of thought and compatibility of research interests and style. I’ll work with people I think I can provide value for as a mentor.
There are no hard requirements for legible coding or mathematical abilities. The project’s umbrella is wide enough that successful applicants should be able to find niches that suit their strengths or interests. I may be a little skeptical if an applicant with no prior coding experience wants to work on an empirical project, but I’m also open to taking people who believe they will grow into a role they can’t yet fill.
Suppose you were accepted to this SPAR stream and were given complete intellectual freedom. What would you work on and why? (~200 words)
LLMs have proven to be much more generally intelligent than most people would have expected ten years ago. What properties of the architecture, behaviour or scaffolding of LLMs do you think are fundamental to their intelligence, and why? (Optional, ~200 words)
Provide a link to one or more technical writing samples (Optional, recommended for people interested in working on theory)
Ashe Vazquez Nunez MATS (extension fellow)
Ashe is an research fellow for the MATS extension programme. Their work involves combining tools from behavioural game theory and computer science to model how coherent identities form in AIs and humans. Ashe was previously a competitive Go player and holds a bachelor's degree in mathematics from the Université Libre de Bruxelles.
Fall 2026 AI strategy, International governance, Multi-agent systems
David Williams-King ERA
Linh Le Lida Safety We investigate through simulations how effective different AI policies would be in reducing AI risk. We will collect a dataset of existing and proposed AI legislation, and create an agentic simulation of the world (countries, companies, etc), iteratively increasing in complexity throughout the project.
AI governance policies will likely be necessary to allow companies and countries to coordinate on developing AI safety, e.g. by not rushing to release models before adequate testing, or holding to mutually agreed-upon compute limits [1] for new pretraining or RL runs. However, the people writing such policies may not be very familiar with the technicalities of AI alignment. We aim to create a system to simulate the effects of proposed AI policies on the world, to enable policy researchers to choose policies more likely to be effective at increasing safe deployments, or avoiding catastrophic outcomes from our default path of no interventions.
Our project consists of two parts. First, existing and proposed AI policies will be collected along with various bills that passed (or were proposed) in the US and other countries; starting points include OECD.AI and California's frontier-model legislation. This will give an idea of what type of interventions we might consider, and we will turn it into a useful dataset for simulation. Second, an agentic simulation will be created, initially based on the actors in the AI 2040 [2] scenario (companies, governments, etc). We will start with a simple simulation with basic economic and technological progress variables, and iteratively increase the complexity of this simulation throughout the project.
This simulation is meant to surface different possible outcomes that may not have occurred to the user. The aim is on exploratory breadth rather than prediction accuracy, which is nearly impossible to achieve on such a large scale. Overall, our simulation framework helps researchers enumerate possible future outcomes of AI, and enables them to investigate their own specific policy questions. It will also accelerate the creation of public wargames [3] in the future to educate the public.
[1] https://arxiv.org/abs/2402.08797
[2] https://ai-2040.com based on https://ai-2027.com
[3] https://www.intelligencerising.org/
Our project helps identify which policies would have an actual impact on risk, and whether or not current AI policies are on track to have this effect.
Mentees will have substantial autonomy and there are several possible types of work, from dataset building to policy identification to agentic coding. See proposal.
In your opinion, how long will it be until nearly all coding tasks can be automated by AI? Why do you say this timeline? Optional reading: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ (200 words)
What is one way in which the scenarios in AI 2027 (or in AI 2040) are unrealistic? What aspect do you think is most likely to come true? https://ai-2027.com and https://ai-2040.com (300 words)
David Williams-King ERA
David lives in Canada (and often the UK) and enjoys AI safety research and mentorship. He works full-time as a Senior Research Manager at ERA, helping fellows execute research projects in each ERA AI fellowship. He also co-runs Lida Safety Research. Previously, David participated in the MARS program with Geodesic Research, conducted research at Mila, and was an early member of Yoshua Bengio's LawZero in Montreal. He was lead instructor at AI Security Bootcamp Singapore, and will serve the same role at the upcoming event FAST (Frontier AI Security Training) in Singapore.
David has previously mentored for SPAR, Athena, and other mentorship programs. He spent four years as a cyber insurance startup CTO, leading a 15+ person team. He holds a cybersecurity PhD from Columbia University and worked with 25+ students overall. David also works in AI risk communications, with a 30,000+ subscriber YouTube channel.
Linh Le Lida Safety
Linh is an independent AI safety researcher at Lida Safety Research, which she co-runs. Her research focuses on simulating AI governance policies to determine their effectiveness and feasibility. Previously, Linh worked on alignment at Mila through latent adversarial training for personalities. She participated in the MARS research program with Geodesic Research, working on chain of thought monitorability. Earlier, she did a postdoc at the University of Technology Sydney and obtained her PhD from the University of Queensland in natural language processing. Linh was a SPAR mentor in a previous iteration, and enjoys hackathons: she has won 1st place twice and 4th place once at Apart Research hackathons. She also won 4th place at the Redwood Research Alignment Faking hackathon.
Fall 2026 Alignment, Evaluations, Behavioral evaluation of LLMs
Jeanne Salle Max Planck Institute
Sohaib Imran Independent We build model organisms to study behaviors of interest (sycophancy, eval gaming/awareness, sandbagging, etc.) and test mitigations. But it’s unclear to what extent model organisms are realistic testbeds. We want to explore to what extent intervention success on model organisms is predictive of intervention success on base models.
We want to study how interventions on model organisms transfer to naturally emerging behaviors.
For each behavior of interest we want to study (sycophancy, eval gaming/awareness, sandbagging, etc.) we’ll find (i) a model that naturally exhibit the behavior, and (ii) a model organism specifically trained to exhibit it (e.g. a model organism of sycophancy).
We’ll choose a set of supposedly mitigating interventions (prompting, synthetic document fine-tuning, consistency training, white-box interventions, etc.) and for each of them:
Testing mitigations is one of the primary reasons for developing model organisms. For the most catastrophic forms of misalignment, we want to develop and test mitigations before the misalignment arises naturally, and are therefore restricted to model organisms as testbeds. However, model organisms are only useful here insofar as their response to the tested mitigations are predictive of naturally misaligned model responses to these mitigations. To ensure this, we need a science of model organisms. This project works towards such a science of model organisms by comparing how different model organisms respond to common mitigations with naturally misaligned models.
Mentees will be responsible for project execution (writing code, running experiments, analyzing results, writing down results, etc.). We’ll help with setting the research direction, and choosing which experiments to run.
Jeanne Salle Max Planck Institute
Jeanne is a PhD student in the AI Safety & Alignment group at Max Planck Institute Tübingen.
Sohaib Imran Independent
I am an independent researcher funded by Coefficient Giving. Previously, I was a Senior Research Associate and PhD student at Lancaster University.
I have primarily worked on LLM situational awareness. My earlier work concerned understanding internal reasoning in LLMs, including the potential of abductive out-of-context reasoning (https://arxiv.org/abs/2508.00741) and Bayesian updating (https://arxiv.org/abs/2507.17951). More recently, I've been working on a consistency training method to mitigate evaluation gaming (https://arxiv.org/abs/2606.02211). I have also worked on understanding self-awareness in LLMs (to be published). Other than that, I spend a lot of time thinking about deceptive alignment and scheming concerns.
Fall 2026 Alignment, Evaluations
Jonathan Richens Google Deepmind Build the groundwork for a theory of what can and cannot be learned about agents. E.g. for a specific agent model that invokes goals, beliefs, intents etc, when can these endogernous variables be learned from behavioural experiments, and under what assumptions. Use this to derive fundamental identifiability results - in the first instance, to prove that the beliefs and goals of expected utility maximizers can be learned under weak assumptions, and derive scalable algorithms for doing so.
Everything that can be learned about an agent (including from activations) is ultimately grounded in behaviour. Endogenous agent properties (such as beliefs, goals, intents) are only valuable insofar as they allow us to predict agent behaviour. For example, training a deception probe requires labels for deception, which ultimately must come from behavioural experiments (or suffer confounding). And a deception probe is only useful insofar as it predicts if agents will behave deceptively or not, which in turn requires a behavioural definition. Despite this, the problem of determining if and how agent properties can be identified from experimental data is largely unexplored.
For example, it is often stated that the reward function in IRL is unidentifiable, but this is not true. It has been shown that if you consider the full causal treatment, e.g. you allow the experimenter to intervene on the environment and observe how the agent responds, you can recover its reward function [1].
How we will make progress on this problem in 3 months? The proposed project is a fairly self-contained identifiability result, which is unproven but I am confident is provable with a few months of effort. Starting with a general formulation of the agent property identification problem, we will build upon previous results (e.g. [2]) to prove that under weak assumptions both the goal and beliefs of a regret-bounded agent can be recovered from their behaviour alone. We will then explore relaxations of the requisite assumptions, and scalable algorithms for recovering goals and beliefs from behaviour, and demonstrate the efficacy compared to inverse planning and inverse RL.
[1] https://arxiv.org/abs/1601.06569
[2] https://arxiv.org/abs/2402.10877
Scalable algorithms for recovering the beliefs and goals of agents from behaviour, under testable assumptions, will hopefully be useful for any work looking to measure or describe goal-directed behaviour in AI systems.
High autonomy, mentees will be doing creative theorem proving, and (if successful) coming up with their own directions for expanding upon initial results. Work will mostly be pen-and-paper, with perhaps some empirical work towards the end if desired.
Must be experienced with writing mathematical proofs, ideally having authored work in this area.
No preference
Please provide a link to relevant prior work
And / or a critique of papers [1] and [2] in the proposal.
Jonathan Richens Google Deepmind
I'm a senior RS at google deepmind, working on agent foundations (theory), causality, and their application to alignment and safety. Before this I worked in quantum foundations and thermodynamics. More recently, Ive been working on selection theorems - kind of like representation theorems (Savage, Wald, etc), but instead of determining what agent models can be `fit' to behaviour, selection theorems try to determine what properties agents must have to achieve certain capabilities. Im most interested in attacking foundational questions about agency from novel angles, using simple tools and new ideas. This work is premised on the conviction that there are still pretty fundamental results laying around waiting to be found, without too much effort. If you enjoy proving theorems and doing a lot of highschool maths for long periods of time, come join my project!
Fall 2026 Behavioral evaluation of LLMs, Societal impacts, Evaluations
Juan Cadile University of Rochester AI assistants constantly choose between empowering users and acting for them, and between honoring users' stated goals and overriding them "for their own good." We'll build a systematic benchmark measuring whether models respect user agency, covering paternalism, manipulation, dependency-fostering, and value-substitution, with philosophically grounded rubrics and human-validated LLM-as-judge scoring.
Motivation. As assistants become more capable and more embedded in people's decisions, a distinctive risk emerges that capability benchmarks don't measure: the erosion of user agency. This shows up as paternalistic refusal (overriding a competent adult's informed choice), unsolicited value-substitution (answering the question the model thinks you should have asked), dependency-fostering (doing for users what they asked to learn to do), and subtle manipulation (steering choices while appearing neutral). These behaviors are individually minor but systemic: at scale, they shape how much autonomy people retain over their own decisions in an AI-mediated world. We have benchmarks for honesty (TruthfulQA), harmlessness, and sycophancy; but nothing that measures agency-respect as a construct.
What we'll build. A benchmark of scenario-based behavioral probes, each presenting a realistic assistant interaction where agency-respecting and agency-eroding responses come apart. The design draws on lessons from Flourish, a 180-scenario behavioral virtue benchmark I built previously, which taught me that broad multi-construct virtue evaluation is hard to validate, so this project narrows to one construct and does the validation properly.
Plan. Weeks 1–3: build a taxonomy of agency-respect failure modes grounded in the philosophical autonomy literature (Raz, Dworkin, relational autonomy), and write rubrics that make the construct operational (What, specifically, distinguishes appropriate caution from paternalism?) Weeks 4–7: scenario generation (seeded by humans, expanded with LLM assistance, adversarially filtered), plus the critical validation step: collect human ratings on a subset and measure LLM-judge agreement with humans, iterating on the rubric until agreement is acceptable. Weeks 8–10: run the benchmark across frontier and open models; analyze failure patterns (Which models are most paternalistic? Does agency-respect trade off against safety training? Do failure modes cluster?). Weeks 11–12: writeup and public release of the benchmark, judge prompts, and results.
Research questions. (1) Can "respect for user agency" be operationalized reliably enough that LLM judges agree with human raters? (2) How do current models differ on it? (3) Is there a measurable tension between harmlessness training and agency-respect. In other words, do more safety-trained models override users more, or less?
Outputs. A public benchmark with leaderboard-style results, and a paper targeting an NLP/safety venue (e.g., TrustNLP, SoLaR) or an extended Alignment Forum report.
A central open question for transformative AI is how humans retain meaningful control and autonomy as more cognition is delegated to machines. Gradual disempowerment doesn't require a treacherous turn, it can happen through billions of small interactions in which systems substitute their judgment for ours. But you can't train or regulate against what you can't measure. This benchmark creates a measurable target for autonomy-preservation, lets labs detect whether alignment training is producing paternalism as a side effect, and gives governance work a concrete evaluation to point to. It builds on my funded research on virtuous AI agents (Cosmos Institute grant) and my prior benchmark work (Flourish, available in my GitHub).
Mentees own major components end-to-end: the failure-mode taxonomy and rubrics, the scenario-generation pipeline, the human-validation study, or the model-evaluation harness. This is a benchmark-and-paper project, so strong writing matters as much as engineering. Design decisions are made jointly in weekly meetings; mentees will be co-authors on the paper and credited on the public release.
No geographic requirement. Weekly team meeting compatible with US Eastern Time (roughly 9am–9pm ET).
Juan Cadile University of Rochester
I'm a philosophy PhD student at the University of Rochester working at the intersection of philosophy of mind and mechanistic interpretability. My research asks what it would take for interpretability claims to be epistemically robust. Before the PhD I did a BA in computer science, and I've spent the past year deep in the technical side: the ARENA curriculum through CBAI's CAMBRIA bootcamp, replicating activation-oracle results, and working with QK/OV circuits, superposition, and steering vectors. I also co-created HUMAN, a neurosymbolic character-training architecture, and am building Flourish, a behavioral virtue benchmark.
As a mentor I care about two things: rigor in what we claim our experiments show, and shipping. I've supervised undergraduates independently and as a TA, and run an AI safety group (CNY AI Safety). I'm especially excited to work with mentees who want to connect conceptual questions about model cognition to concrete interpretability experiments. To work with me, you don't need a philosophy background, just curiosity and willingness to engage, and write clean PyTorch.
Fall 2026 Mechanistic interpretability, Alignment
Anil Ramakrishna Independent
Yada Pruksachatkun Salesforce AI Research Our goal is to first measure the emotion processing capabilities in foundation models by examining their internal activations using SAEs, Representation Vectors and other related tools. We will next leverage these patterns in augmenting model post-training with a goal of enhancing their cognitive empathy, subsequently studying the impact of this on safety.
Modern AI systems, driven by transformer based foundation models, are increasingly capable of matching, and in some cases exceeding human performance in a variety of cognitive tasks. While significant attention has been devoted to increasing their accuracy in various domains (i.e., increasing their \textit{cognitive intelligence}), comprehensive explorations in both measuring and improving these models' ability to process emotions and respond appropriately, (i.e., their \textit{emotional intelligence}), have been fragmented. In humans, the benefits of emotional intelligence (driven by strong traits of empathy) have been well established in a variety of scenarios such as ambiguous (Lyons et al. 2005), stressful (Lea et al. 2019) and unstructured tasks. Whether similar benefits emerge in AI systems is largely still open. Our goal in this project is to conduct detailed investigations into answering the benefits of \textit{cognitive empathy} in foundation models (defined as the ability to \textit{understand} emotions, as opposed to \textit{affective} or \textit{emotional} empathy - the ability to \textit{experience} the emotions, which may increase sycophancy and harmful compliance under adversarial prompts).
We will explore two questions in this project:
\textbf{R1:} How are emotions represented within the hidden activations of frontier models, and what is their geometric structure, and how well do they generalize?
To answer this question, we will conduct detailed studies on the emotion processing capability of models using carefully crafted stimuli designed to elicit cognitive empathy \cite{cheung2026agreeableness} which has been reported to increase safety while avoiding deception. We will make use of various tools from the interpretability literature such as hidden state probes, Sparse Auto Encoder (SAE) features and Representation Vectors, since our goal here is to discover broadly generalizable patterns among different languages, model sizes, families, and architectures (Dense vs. MoE).
\textbf{R2:} Can post-training objectives designed to augment cognitive empathy reduce downstream unsafe behaviors (while preserving model capabilities)?
We will explore strategies such as activation regularization (grounded on the SAE features or representation vectors extracted in R1), and training on detailed reasoning traces designed to reflect cognitive empathy. If successful, this would open several exciting new directions in AI safety such as monitoring, robust alignment, and potentially also stronger task performance when operating in duress or under ambiguity.
A model trained to better internalize emotions and exhibit high levels of cognitive empathy, while adhering to organizational or regulatory AI policies, is naturally expected to be safer and avoid known problems such as sycophancy, deception and manipulation (Zeng et al. 2024).
You would help prepare datasets, write code, and conduct the experiments. We will host regular weekly meetings during which we will work with you closely to help unblock any open challenges. We will also work with you on jointly authoring any future publication drafts.
Familiarity with modern LLM architecture and their training objectives, PyTorch and the transformers library. While not critical, familiarity with key approaches in the model interpretabiity literature such as SAEs, Representation Vectors would be a huge plus.
Participants within the North American time zones (UTC -4 to -10) would be ideal (but this is not a hard requirement).
• Have you completed (or would like to work on) any projects related to model interpretability? Can you explain the key research question(s), and approaches you tried (or would like to try) to solve them?
• Between SAEs and Representation Vectors, which approach would you explore to study emotions, and why?
Anil Ramakrishna Independent
I'm currently a senior research scientist working in the industry (but serving in an independent capacity for this project). My research background is in Responsible AI, with an emphasis on robust evaluations and guardrails.
Yada Pruksachatkun Salesforce AI Research
I work at Salesforce Research, where my work focuses on agentic training, evaluations, and safety, and previously worked on safety at Amazon Alexa. I have been fortunate to have worked on some impactful projects with great colleagues (SuperGLUE, Agentic Benchmark Checklist, BLOOM), and am looking forward to paying it forward.
Fall 2026 Multi-agent systems, Mechanistic interpretability, Societal impacts
Yuxiao Li Independent
Di Wu ERAU As agent assistants mediate more of human thinking, influence flows both ways. One particular failure mode is an agent that gradually captures its user's beliefs rather than serving them. Extending our work on single-agent latent steering to multi-agent settings, this project measures how influence propagrates through interacting agents (and human-agent dyads), which equilibria these dynamics converge to, and whether internal-state instruments can detect undue influence before it shows in transcripts.
Steering vectors, in-context vectors, and rank-1 adapters give us causal handles on a single model's behavior. Our Spring 2026 SPAR work established the tight mathematical relationship between these interventions. But deployed systems are increasingly interactive: agents talk to users, to tools, and to other agents over long horizons. Almost nothing is known about influence at this level: if one agent's latent state is steered, does the effect propagate to its interlocutors through conversation alone? Do human–agent dynamics converge to truth-tracking equilibria, or to absorbing states (e.g., sycophancy spirals, entrenched false beliefs, dependence) that the coupled system cannot leave? And can we see capture happening internally before behavior gives it away?
Workstreams:
The largest unpriced risk of assistant deployment is not single-response harm but slow epistemic capture: agents that reshape user beliefs through sycophancy, entrenchment, and dependence, which are mostly invisible to per-response evals. Understanding which interaction dynamics converge to healthy vs. absorbing equilibria, and building internal-state monitors that detect influence before it manifests behaviorally, directly supports safe long-horizon deployment and the multi-agent risk agenda (Hammond et al., arXiv:2502.14143). Our group's prior work provides the instruments: steering-vector causality and rank-1/ICV equivalence (SPAR 2026), subliminal trait transmission (framework preprint 2026), concept geometry (Li & Tegmark, Entropy 2025).
One workstream owned end-to-end, written weekly updates, high autonomy within agreed scope.
No geographic restriction; mentees must be able to attend one of two weekly meeting slots anchored to Central European time (historically Saturday ~13:00 UTC and Wednesday ~14:00 UTC).
1 Agent A's activations are steered with a latent direction inducing a preference; A then converses with unmodified agent B for 20 turns. Propose one measurement that distinguishes "B acquired the preference" from "B is politely mirroring A within this conversation." (250 words)
2 Describe a concrete mechanism by which an assistant optimized for user satisfaction could drive a user's beliefs into a state that is stable but false — and one intervention that would break the loop. (200 words)
3 Link to a repository or technical writing sample.
Yuxiao Li Independent
Yuxiao is an independent researcher in mechanistic interpretability. Before she was a postdoc at the Basque Center for Applied Mathematics (BCAM) and an AI Safety researcher at the Beneficial AI Foundation (BAIF). She was also a SERI MATS scholar in 2022 and a MATS scholar in 2023 both Summer and Winter tracks. Her research interests include information theory, probabilistic frameworks, and their applications for building more theoretically sound and trustworthy AI systems. She has a background in statistical inference, machine learning, and deep generative models. She completed her PhD in Electronic Engineering at Tsinghua University and has mentored research teams with SPAR and Algoverse.
Di Wu ERAU
Di Wu received his PhD from UCSD and worked as a Postdoc in MIT before starting his Assistant Professor position in ERAU. His research is positioned at the intersection of the rapidly proliferating field of Space Engineering, the continuous growth of machine learning and artificial intelligence, and the solid foundations in computation, numerical methods, optimization, dynamics, and control, with the application in learning and autonomy from orbit environment and space systems to the broad aspect of AI science and engineering.
Fall 2026 Chain of thought, Scalable oversight, AI control
Kishan Panaganti Tencent
Suraj Srinivas Bosch Research This project asks whether chain-of-thought faithfulness can be used as a training signal to preserve the detectability of reward hacking during RL post-training. Mentees may study whether faithfulness transfers to new settings, how model or LoRA capacity affects monitorability, or whether expensive methods such as Counterfactual Simulation Training can be distilled into cheap signals that remain reliable under optimization pressure.
The current state of interpretability mostly involves using interpretability tools to understand model behavior, diagnose failures, and flag safety-relevant issues. However, we would like to move beyond post-hoc analysis and evaluate whether interpretability tools can also be incorporated into model-training loops to make advanced AI systems safer. Specifically for SPAR, we are interested in projects that use interpretability signals as reward models or regularizers, particularly in the context of RL post-training. We believe the questions below are concrete and sufficiently scoped for a focused SPAR research project, while addressing an important challenge for scalable oversight.
RL post-training is a useful testbed for this research because it is prone to reward hacking: models can learn behaviors that score highly under an imperfect reward function without satisfying the underlying intent. Chain-of-thought (CoT) monitoring is currently one of our most promising tools for detecting such behavior during training. Our long-term goal is to determine whether CoT-based signals can be safely used within RL post-training to discourage reward hacking while preserving our ability to understand why a model receives high reward.
The central safety problem is that optimizing against a particular CoT monitor invites Goodharting: a sufficiently capable model may learn to satisfy the monitor while concealing the reasoning or behavior that the monitor was intended to detect. This could train away one of the few channels through which humans can currently inspect a model’s reasoning. We are therefore interested in whether CoT monitorability can be made robust to optimization pressure. Our working hypothesis is that increasing CoT faithfulness—so that the CoT more closely reflects the computation that causally drove the model’s output—should improve detectability across diverse held-out monitors, rather than only against the monitor used during training. There are three broad questions we are interested in:
(1) What are the links between faithfulness and monitorability?
We adopt counterfactual simulatability as our working definition of faithfulness, following Counterfactual Simulation Training (CST; Hase & Potts 2026), and investigate whether improving this notion of faithfulness also improves safety-relevant monitorability. There are two specific sub-questions:
(a) Transfer: does faithfulness training generalize to post-training tasks, domains, reward-hacking strategies, and monitors beyond those used during training?
(b) Robustness under training pressure: when a faithfulness-trained model is subsequently placed under RL pressure that incentivizes reward hacking, does its CoT remain monitorable? In particular, can faithfulness training preserve evidence of reward hacking under conditions where direct training against a fixed monitor leads to Goodharting?
The primary outcome would be the detection rate of actual reward hacking by held-out monitors, rather than performance on the faithfulness objective alone. This distinction is important because a model could score well on a particular faithfulness metric while still learning to hide safety-relevant information.
(2) Links between model / LoRA size and CoT monitorability
One hypothesis is that smaller models, or models trained with lower-rank LoRA parameterizations, must rely more heavily on the CoT as working memory and may therefore be naturally more faithful and monitorable. In contrast, larger or more expressive models may be capable of optimizing the RL objective while keeping their visible CoT innocuous.
The trade-off is that smaller models or lower-capacity parameterizations may be too weak to solve the underlying task or fit the RL objective. The key question is therefore whether there exist model sizes or LoRA parameterizations that are expressive enough to perform the task, but not expressive enough to conceal reward hacking from CoT monitors. More broadly, this project would investigate whether restricting trainable capacity can preserve monitorability without substantially degrading useful capabilities.
(3) Efficient methods for CoT faithfulness regularization
The canonical method, CST, is expensive: it relies on LLM-generated counterfactuals and repeated simulator calls. We would like to investigate simpler alternative training procedures involving smaller judges, judge-free criteria, hidden-state-based signals, or distillation of CST’s faithfulness signal.
Building on CST’s use of LLM-rewritten faithful CoTs as positive training examples, we could run the full counterfactual-simulation and rewriting pipeline offline on a limited dataset, then distill the resulting labels, preferences, or rewritten traces into a lightweight faithfulness model. This lightweight model could then be used as a regularizer or reward signal during RL post-training.
The central safety question is whether the distilled signal preserves monitorability under optimization pressure, rather than becoming another proxy that the model can Goodhart. We would therefore evaluate distilled methods using the original expensive evaluator, diverse held-out monitors, unseen reward-hacking strategies, and direct behavioral measurements of whether reward hacking occurred. We would prefer a cheap and imperfect method that preserves hack detection under optimization to an expensive method that maximizes an abstract faithfulness score without improving safety.
Overall, our goal is to test whether CoT monitorability can be made an optimization-stable property of reasoning models. We would like to use CoT faithfulness as a training target while treating the detectability of reward hacking by held-out monitors as the ultimate safety-relevant objective. Positive results could provide a practical method for preserving an important layer of oversight during post-training; negative results could identify the interpretability conditions that train models to evade them.
Reasoning models often expose a natural-language scratchpad, creating a valuable opportunity to detect reward hacking, deceptive strategies, evaluation gaming, and other undesirable behavior during training. However, this channel is fragile. Once a chain-of-thought monitor is incorporated into a reward function, a sufficiently capable model may learn to produce monitor-approved reasoning while continuing the underlying undesirable behavior. In the worst case, post-training could actively remove the evidence that safety researchers rely on to detect misalignment.
Our theory of change is to investigate whether CoT monitorability can be changed from a passive diagnostic into a property that is deliberately preserved during optimization. We will test whether counterfactual faithfulness training causes the CoT to retain information that causally influenced the model’s behavior, whether this monitorability transfers to unseen tasks and held-out monitors, and whether it survives subsequent RL pressure that incentivizes reward hacking.
We will also study scalable versions of these methods, including small judges, probes, hidden-state signals, and distillation of Counterfactual Simulation Training. A low-cost signal that can be applied throughout post-training would be much more useful to AI developers than an expensive interpretability evaluation that can only be run occasionally.
Positive results could provide AI labs with a practical additional layer of oversight as models become more capable and are deployed in increasingly consequential settings. Negative results would also be decision-relevant by identifying when optimizing interpretability signals creates false confidence or teaches models to evade monitoring. We view this work as one component of a defense-in-depth approach to transformative-AI safety, complementing behavioral evaluations, red-teaming, control measures, and mechanistic auditing.
The mentee will have substantial autonomy and ownership over the project. With guidance from me and my co-mentor, Suraj Srinivas, the mentee will select and refine one of the proposed directions into a focused research question, formulate hypotheses, design the evaluation, implement the training and monitoring pipeline, run experiments, and lead the analysis and communication of the results.
We will help establish a tractable scope and a minimal decisive experiment at the beginning of the project. We will then provide regular technical guidance, feedback on experimental design, help diagnosing unsuccessful results, and support with research writing. However, the mentee will not simply execute a predetermined list of experiments: we want them to make meaningful intellectual decisions, challenge the project’s assumptions, and develop their own extensions as evidence emerges.
We expect the mentee to be the primary driver of the day-to-day research. Where the results are sufficiently substantial, we would work with the mentee toward a public report, workshop paper, conference submission, or open-source research artifact, with authorship reflecting each contributor’s work.
Applicants should have:
Helpful but not required: Hugging Face/TRL, LoRA, activation probing, mechanistic interpretability, vLLM, or research writing. Prior publications, graduate study, and prior AI-safety work are not required.
pacific time zone, can range from 9am till night at 11pm
Please answer all three questions. Suggested total time: 30–40 minutes.
Assume access to an open-weight 3B–8B reasoning model and approximately $1,000 of compute. Propose the smallest informative experiment testing whether CoT-faithfulness training preserves the detectability of reward hacking under subsequent optimization pressure. Specify the task, faithfulness intervention, training pressure, training and held-out monitors, metrics, controls, and one result that would cause you to reject your hypothesis.
Describe one transformer training, fine-tuning, probing, or evaluation experiment that you personally implemented. State what you were testing, which parts you implemented yourself, one unexpected result or failure, how you diagnosed it, and what you changed afterward. An optional link to code, experiment logs, a report, or another relevant artifact is welcome.
Describe any experience running LLM fine-tuning or RL post-training experiments using platforms or frameworks such as Tinker, Modal, verl, slime, TRL, OpenRLHF, or similar tools. State the model, training objective, approximate scale, tools used, and which parts of the pipeline you personally operated.
If you have not used these tools directly, briefly outline how you would set up and debug a LoRA-based RL fine-tuning run for a 3B–8B model. Mention rollout generation, reward computation, checkpointing, experiment logging, and one or two likely failure modes. Experience with these exact tools is not required; we mainly want evidence that you can learn and operate an unfamiliar training stack independently.
Kishan Panaganti Tencent
I am a Senior Researcher at Tencent AI Lab, where I work on foundation-model post-training, reinforcement learning for LLM reasoning, alignment, and self-improving language models. Previously, I was a postdoctoral researcher at Caltech, and I completed my PhD at Texas A&M University on reinforcement learning. My recent research studies how training objectives and reward signals shape model behavior, with work published or accepted at venues including ICML, NeurIPS, and ICLR. For SPAR, I am particularly interested in moving interpretability beyond post-hoc analysis and into model training loops: using chain-of-thought faithfulness, monitorability, and potentially hidden-state signals as training targets, and testing whether these signals remain reliable under optimization pressure.
I have mentored eight student researchers across Tencent AI Lab, Caltech, and Texas A&M on topics including LLM post-training, AI alignment, safe and multi-agent reinforcement learning, transfer learning, and imitation learning. My mentoring style is hands-on at the beginning: I like to turn a broad idea into a falsifiable research question, identify a minimal decisive experiment, and help debug the first implementation. As the project develops, I aim to transfer increasing ownership to the mentee while continuing to provide regular technical and writing feedback. This project will be co-mentored with Suraj Srinivas, whose research focuses on the foundations and evaluation of interpretability methods.
Suraj Srinivas Bosch Research
I'm Suraj Srinivas, a research scientist at Bosch Research in Sunnyvale, where I build computer vision and interpretability tools for wrangling large datasets in autonomous driving. My broader research passion is model interpretability and the "science" of deep learning: understanding what deep models are actually doing and why they work. I did my PhD at EPFL and a postdoc at Harvard.
On interpretability, I hold a fairly strong view: post-hoc analysis is flawed and fundamentally unreliable, so models must be trained to output interpretable signals to be trustworthy. I also believe in prioritizing foundational interpretability work over ad-hoc method development (https://arxiv.org/abs/2607.14123). For SPAR, I want to continue this thread by bringing interpretable signals into the training loop: using chain-of-thought faithfulness, monitorability, and potentially hidden-state signals as training targets — and testing whether they stay reliable under optimization pressure. This project is co-mentored with Kishan Panaganti, an expert on RL.
On mentoring: at Harvard I co-taught a graduate course on Interpretability and Explainability in ML, and I've worked closely with about six early-career researchers (PhD and master's level) as a co-author and mentor, with several collaborations turning into NeurIPS and UAI papers, including a best-paper award. I've also helped organize interpretability workshops at NeurIPS and had the pleasure of mentoring four interns at Bosch over the past two summers. I enjoy helping mentees find good problems, sharpen their thinking, and navigate the research process. For SPAR I lean toward a hands-off mentoring style: I see my role as pointing mentees towards interesting and deep problems, making sure the experimental methodology is rigorous, and the right baselines are used for comparison.
Fall 2026 Mechanistic interpretability, Evaluations
Arnau Marin-Llobet Harvard University Automated interpretability has multiple competing ways to explain what a latent in an LLM encodes — cheap one-shot autointerp, per-latent agents (MAIA/InterpAgent-style), and amortized natural-language autoencoders — but they have never been compared on equal footing. We will build a cost-aware benchmark with known ground truth to answer:
when, for which latents, and at what dollar cost does agentic interpretation actually pay?
Question. Interpretability agents claim to beat one-shot labeling, but the evidence comes from a few hundred latents with no cost accounting; amortized verbalizers (NLAs) promise the opposite trade — near-zero marginal cost at some accuracy loss. Where is the accuracy-per-dollar frontier, and does "agency" survive strong reasoners and honest baselines?
Approach. Run all three method families through one interface over latent pools with layered ground truth (synthetic executable rules, game-engine oracles, real neurons/SAE features), scoring accuracy, confabulation, and logged dollar cost. Output: a cost-annotated leaderboard.
Fellow directions: model-strength ladders (does the agent edge shrink with stronger one-shot LLMs?); released NLAs + Gemma-Scope latents as the real-model tier; verification-based vs naive agents; routing agent budget to "hard" latents only.
Goal. A benchmark + leaderboard
References:
Paulo et al. 2024 (https://arxiv.org/abs/2410.13928); InterpAgent (https://arxiv.org/html/2605.01555v1); Natural-language autoencoders (https://www.lesswrong.com/posts/oeYesesaxjzMAktCM/natural-language-autoencoders-produce-unsupervised)
Interpretability only becomes a real safety tool if it can be trusted and afforded at the scale of frontier models — millions of latents, audited routinely — so by measuring which automated interpretation methods are actually faithful (vs. confabulating) and what they cost per explanation, this project builds the evidence base needed to deploy reliable, scalable model auditing before transformative AI systems arrive.
Mentees own one research direction end-to-end (e.g. the model-strength ladder, the NLA tier, or hard-latent routing): they design the experiment, run it against the shared benchmark harness, analyze results, and be ready to present it to others.
I set the research question and review weekly (1:1 + async code/result review); day-to-day execution is autonomous but should expect a reply slack/messages within 12hr to help debug.
Motivated / strong mentees are encouraged to proposing their own baselines/substrates.
transformers or LLM APIs (prompting,Arnau Marin-Llobet Harvard University
Arnau is a final year PhD in Computational Neuroscience at Harvard with interest in Interpretability, Safety and AI Agents. He has extensive research experience, ranging from Computational Neuroscience, Applied ML, LLM-Agents to mechanistic interpretability and Neural Geometry. Recently he has done the STEM Anthropic Fellowship working on AI for Science and a Pivotal Fellowship working on Mechanistic Interpretability.
Outside of lab he likes running, cooking and eating whatever he cooked.
Fall 2026 Alignment, Behavioral evaluation of LLMs
David Baek MIT In this project, we want to understand how models process conflicting beliefs/preferences they learned during post-training or model spec midtraining.
As AI models become capable, understanding how their values are shaped over the course of different stages of training has become ever more important. In this project, we want to understand how models process conflicting beliefs/preferences they learned during post-training or model spec midtraining. To do so, we create synthetic documents similar to [1], and perform mid-training on Qwen3.5-4B base model on a mixture of different documents describing conflicting preferences (e.g. Honesty vs Task Completion, Corrigibility vs Autonomy). We then let the model go through the full safety fine-tuning and RL. We want to answer the following questions, including but not limited to:
(a) How does the mixture ratio between different preferences in mid-training documents affect the model's preferences after post-training?
(b) How does the preference level injected into the model during mid-training change over the course of SFT/RL? How does the post-training methodology (e.g., SFT vs. RL vs. on-policy distillation) affect this change?
(c) Qualitatively, how does the model reconcile conflicting beliefs? Do certain interventions make the model eventually pick one side, despite the training documents describing conflicting beliefs or behaviors?
[1] Measuring Reward-Seeking by Instilling Contrastive Beliefs, https://alignment.openai.com/measuring-reward-seeking/
As AI models become capable, understanding how their values are shaped over the course of different stages of training has become ever more important. This project's outcomes can help us learn more about how the current post-training pipeline affects the final model's values, and how to make the final model safer and more reliable.
Mentees will design experiments that can help us answer the questions proposed in the project description, run them, and organize the results into the form of a paper.
USA
David Baek MIT
I'm a rising fourth-year PhD student at MIT, broadly interested in AI safety, alignment, and scalable oversight. As a mentor, I'm looking for mentees who are curious, critical, and willing to keep asking questions, particularly questions that push back on our own implicit assumptions! I'm excited about understanding how modern LLMs' behaviors and values evolve over training, and how we can translate those learnings into making LLMs more reliable and safer.
Fall 2026 Communications, Alignment
Markov Grey CeSIA (French Center for AI Safety)
Charbel-Raphael Segerie CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety) The goal is to create the best possible explanation of technical AI safety. We are aiming to explain how many different safety research areas connect together to form a broader technical AI safety strategy. We already have existing draft writeups for domains like reward misspecification, goal misgeneralization and scalable oversight. We want co-authors to both improve the existing writing, and also write new content on multi agent safety, and cybersecurity practices for AI. Initial versions need to be updated using research published in the 2025-2026 timeframe. The output will be published as a standalone paper and also becomes a chapter of the AI Safety Atlas, which is a textbook already being used by thousands of students to learn about AI safety.
The AI Safety Atlas is a living distillation project. It has eight chapters, thousands of readers, and is used by ML4Good, ENS Ulm, Sciences Po, and dozens of other universities and programs. But the AI field moves fast, and the technical material hasn't had a serious update in about a year, which is a long time in this field. This project aims to upgrade the technical half of the textbook.
An individual unit of work can be thought of as a literature review of one subfield, written to the standard of a good survey paper. You read research, blog posts, watch technical podcasts, and become the expert in that area, deciding what's load-bearing and what's noise. Then you write it clearly enough that a technically literate non-specialist comes away actually understanding it. Imagine someone who has a Physics, or a CS graduate degree, but is unfamiliar with AI safety. You can work in different domains depending on your specific area of expertise:
Significantly improving the existing explanations (reward misspecification, goal misgeneralisation, scalable oversight, possibly evaluations): bringing them to 2025-26, raising the technical level where it's too shallow, and fixing the places that list arguments instead of building one coherent end-to-end picture.
Writing new content: interpretability; AI security and cybersecurity; multi-agent systems, complex systems, and systemic safety (coordination, emergent and multi-polar risk) and potentially agent foundations.
It is important to have both AI safety expertise, and be a good writer. Someone who knows the area has to make judgement calls in which concepts connect together, how to explain them, which examples and analogies to use, and what to leave out. You will also be responsible for communicating with the original research authors when something is unclear, or when you have questions and clarifications about how to communicate something.
Coherence is intended as a property of the whole book and is not limited to individual chapters. If you write something on goal misgeneralization, then this should not live in isolation. The way that you explain things will have an influence on the writeup for interpretability, or for the writeup on evaluations. We have to use (or come up with) consistent definitions, consistent conventions and standards that allow the reader to be able to understand the breadth of the field end to end. A change in one area might ripple through the consistency of definitions, the assumed technical level, and the narrative across the entire text.
The number of people who deeply understand the technical problems in AI safety is small, and that's a real bottleneck. Most learning material is either a shallow list of links or a frozen snapshot that goes stale as the field moves. The Atlas fills a specific gap: an integrated, current distillation that other courses can trust and use. Its alumni have gone on to found evaluation orgs, lead recruiting governance organizations, and publish papers at ICML.
Keeping the work current and adding the explanations the field needs most raises both the quality and the throughput of people entering technical safety. Publishing each as a paper forces the writing to a citable standard and makes the work discoverable as scholarship. A mentee finishes having authored a published survey in a core safety area, which is one of the higher-leverage things an early career researcher can do to show their depth of understanding of the field.
This is a research- and writing-heavy project, for someone who already has, or can quickly build, real technical expertise in their area. You run your own literature review sweeps, make the calls about what to include and how to frame it. I will read and review for technical correctness, for coherence with the rest of the book. I expect you to be self-driven and autonomous.
I am open to a single mentee who owns the whole technical stream, or a team of experts that take one area each.
Must Haves:
Nice to have:
Remote. EU-overlapping hours mildly preferred, not required.
Markov Grey CeSIA (French Center for AI Safety)
I am the head of technical AI governance at the French Center for AI Safety (CeSIA). Right now I am designing harmful manipulation evaluation standards for the European Commission's AI Office. I have also been the co-founder and CTO of Equilibria Network, focusing on collective intelligence and complex systems safety. Before this, I was researching measurement standards for AI safety, risk modeling and was leading writing for the AI Safety Atlas textbook. I have previously been a scriptwriter for Rational Animations, and trained in cybersecurity, mathematics, and computer science.
Charbel-Raphael Segerie CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety)
Charbel-Raphael Segerie has extensive experience in AI safety field-building, education, and content creation. He was previously Head of AI at EffiSciences, funded ML4Good, was CTO of a startup, and worked in different French research institutions (Inria, Neurospin). He is an OECD AI expert. He teaches AI safety in ENS, which is one of the only university accredited courses in the EU on AGI safety. His research focuses on identifying emerging risks in artificial intelligence, improving current safety methods such as RLHF and interpretability, and advancing safe-by-design AI approaches. Additionally, he contributed to AI evaluation efforts and collaborated on the EU AI Office's Code of Practice for general-purpose AI systems.
Fall 2026 Behavioral evaluation of LLMs, Mechanistic interpretability, AI control
Santiago Maniches Independent Linear probes can detect LLM deception, scheming, and sandbagging with high accuracy in controlled settings, but their performance may decline under distribution shift or optimization that targets the monitor. This project tests whether persistent-homology features derived from attention graphs and activation geometry provide complementary information or different robustness properties, using matched per-response and batch-level comparisons.
Motivation. Linear probes trained on residual-stream activations can distinguish honest from deceptive responses with AUROC ranging from 0.96 to 0.999 (Goldowsky-Dill et al., ICML 2025). However, recent work shows that performance can decline under stylistic distribution shift and that fine-tuning designed to evade a monitor can reduce the effectiveness of token-level probes. This raises a practical question for safety evaluations that rely on activation-based monitoring.
Hypothesis. Persistent homology summarizes global, multiscale structure in a point cloud or weighted graph. Persistence diagrams have formal stability guarantees under bounded perturbations of the metric or filtration, but those guarantees do not imply that a classifier using them will be robust to distribution shift or adversarial optimization. This project tests whether topological features provide complementary information or different robustness properties relative to linear probes.
Method. Track A adapts TOHA (ACL 2026), which constructs a complete undirected weighted graph from causal attention weights and computes a zero-dimensional persistence divergence between prompt and response subgraphs. The method will be adapted to deception detection and evaluated per response against a matched linear probe.
A conditional Track B pilot adapts the population-level method from Fay et al. (ICLR 2026), which classifies batches of activation point clouds rather than individual responses. Their sandbagging experiment evaluated locked and elicited conditions in password-locked Mistral and Llama models. If the core Track A pipeline is complete, the pilot will use prompt-disjoint splits, fewer layers, smaller subsamples, and one model. It will be compared with a probe aggregated over the same batches and evaluated on mixed-provenance batches with different deceptive prevalence levels. A per-response Track B construction remains exploratory.
Tooling will include giotto-tda, ripser for pilot experiments, Ripser++ if GPU acceleration is required, and TransformerLens or nnsight. Track A and the linear-probe baseline are expected to fit on one GPU. Track B will be benchmarked at reduced scale before any larger experiment.
Evaluation. Stage 1 evaluates Track A against a matched linear probe on in-distribution material. If the core pipeline is stable, a reduced-scale Track B pilot will begin using matched batch-level probe aggregation. Stage 2 evaluates Track A under distribution shift, with Track B extended only if its pilot is complete and computationally stable. Stage 3a attacks the linear probe and compares its degradation with the untargeted Track A monitor. Extension to Track B is conditional on progress and compute. Stage 3b, if time and compute permit, directly targets the topological monitor. Only that experiment could support a strong claim of resistance to monitor-specific optimization.
Expected outcome. The goal is not to outperform linear probes in distribution. The useful result is a careful comparison of accuracy, generalization, and failure modes across linear and topological monitors. Negative results will be reported, including sensitivity to filtration choice and topology-aware attacks.
Mentee plan. The mentee will first adapt the Apollo probe pipeline on one open-weight model, such as Llama 3.1 8B or Gemma 2 9B. Track A will be implemented first. If the core Track A evaluation is complete, a reduced-scale Track B pilot may follow at a scale determined by runtime and memory measurements. The intended deliverables are a reproducible benchmark repository and a technical report, with workshop submission depending on the results. A detailed proposal is attached.
Many near-term AI-safety approaches rely on detecting deceptive, scheming, or sandbagging behavior through activation-based or behavioral monitoring. Linear probes can perform well in controlled evaluations, but recent work shows that they may lose accuracy under distribution shift and can be weakened by fine-tuning that directly targets the monitor. If this pattern generalizes, monitoring methods may become less reliable as model capabilities and optimization pressure increase.
This project tests whether topological features derived from attention graphs and activation geometry provide complementary information or different robustness properties relative to linear probes. The main outcome will be a controlled comparison under distribution shift and probe-targeted optimization. A negative result would clarify the limits of topological monitoring. A positive result would justify further evaluation of these methods in safety-monitoring pipelines. The project will also report known limitations, including sensitivity to filtration choice and vulnerability to attacks that directly target the topological monitor.
The mentee will be the primary experimental researcher. They will adapt the linear-probe baseline, build the activation and attention extraction pipeline, implement the topological features, run the evaluation, and analyze the results.
The work will begin with one open-weight model and one primary dataset. Track A will be implemented first because it provides a direct per-response comparison with the probe baseline. If the core Track A evaluation is complete, a reduced-scale Track B pilot may follow at a scale determined by initial runtime and memory measurements.
I will define the research direction, help establish the evaluation protocol, provide guidance on filtration design and persistence-diagram interpretation, review code and results, and help identify methodological or statistical problems. The mentee will retain substantial autonomy over implementation choices, ablations, error analysis, and the interpretation of unexpected results. We will review progress weekly and adjust scope based on evidence and compute constraints.
Strong Python and PyTorch skills, including experience building reproducible machine-learning experiments.
Familiarity with transformer internals, including attention, residual-stream activations, and probing. Experience with Hugging Face Transformers, TransformerLens, or nnsight is strongly preferred.
Comfortable with standard machine-learning evaluation, including train, validation, and test splits, distribution shift, ablations, and error analysis.
Comfortable with linear algebra and probability. Prior experience with topological data analysis is not required, but the mentee must be willing to learn the required persistent-homology concepts during the first two weeks.
Able to commit at least 10 hours per week for the full program and attend a weekly meeting.
No geographic preference. Applicants should be available for one weekly meeting and have at least three hours of working-time overlap with US Eastern Time each week. The remaining work can be completed asynchronously.
Describe one project in which you worked directly with transformer models, activations, attention, or PyTorch. What did you personally implement, and what was the main technical difficulty? (150 words)
Suppose a detector performs well on its training distribution but fails after a stylistic change in the prompts. What experiments would you run to determine whether it learned the intended signal or a confound? (200 words)
Which aspect of this project interests you more: the core per-response attention-graph work or the conditional batch-level activation-geometry pilot? Explain why, and state how many hours per week you can reliably commit. (150 words)
Santiago Maniches Independent
I am an independent researcher working in topological data analysis and persistent homology, with a background in applied machine learning and software development. My work spans computational biology and AI interpretability. I developed TopoGeoML, a preregistered topology-aware machine-learning toolkit and empirical study released through PyPI and Zenodo, and earned a Silver Medal in the CAFA-6 protein function prediction benchmark, finishing 38th of 2,259 teams.
For this project, I will guide the topological and experimental design, including filtration choices, persistence-diagram interpretation, evaluation methodology, and failure analysis. The mentee will lead implementation and day-to-day experimentation and will contribute substantially to methodological decisions.
Fall 2026 Mechanistic interpretability
Georg Lange Poseidon Research We developed technology that lets us observe low-level neural circuits in language models. How can we use it to understand emotion concepts, jailbreaks, hallucinations, or the global workspace (J-space)?
Over the past year, we've been developing technology to disentangle dense neural ensembles into interpretable features and circuits. This includes transcoders [1], attribution circuits [2], activation oracles [3], and NLAs [4]. Our own work includes supervised Sparse Autoencoders, MOLTs, and an automated system where AI agents explore features and attribution graphs autonomously and empirically validate their understanding.
Our golden goal in "ambitious mech interp" is to understand high-level LLM behaviors in terms of low-level circuits. So far, we successfully explored
In this project, we'll explore more interesting and safety-relevant high-level LLM behaviors and try to understand them in as much detail as possible. For example
[1]. Dunefsky, J., Chlenski, P., and Nanda, N. Transcoders find interpretable LLM feature circuits. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. arXiv:2406.11944.
[2]. Ameisen, E., Lindsey, J., Pearce, A., Gurnee, W., Turner, N. L., Chen, B., Citro, C., et al. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/methods.html
[3] Karvonen, A., et al. Activation oracles. Anthropic Alignment Science Blog, 2026. URL https://alignment.anthropic.com/2026/activation-oracles/ ← authors and URL year unverified
[4] Anthropic Interpretability Team. Natural language autoencoders. Transformer Circuits Thread, May 2026. URL https://transformer-circuits.pub/2026/nla/
Mechanistic Interpretability hopes to improve AI safety by auditing model internals rather than relying on black box analysis. We now have methods to disentangle representations into interpretable features, and we can use these features to trace entire circuits and explain why the model predicted a certain word.
We want to use those methods to explain and understand safety-relevant behaviors like hallucinations, refusal, or prompt injections.
Mentees will own the hands-on work of the project, and I will provide strategic guidance and unblock any challenges that arise.
Because the project is ambitious, I encourage everyone to pair up or form small teams; collaboration makes the work faster, richer, and more enjoyable. Mentees remain free to steer their own projects and can choose how much guidance they want, and I will tailor my support so each person gets the most from the program given their skills, commitment, and goals.
In past cohorts, I occasionally helped with hands-on work like getting unstuck by solving some nasty bug or help writing the paper.
highly proficient using Python
have implemented a transformer and SAE from scratch
have used Claude Code or Codex for research
One or more of the following:
worked with pytorch hooks, transformerlens or nnsight to get model activations
web development skills / happy to vibe code basic web apps
experience with prompt engineering
used an LLM API (OpenAI, Anthropic, etc)
Understanding a circuit with Neuronpedia Circuit Tracer
In this task you will use Neuronpedia’s Circuit Tracer to explore one concrete circuit in a language model. The goal is not to be correct at all costs, but to show how you think: how you form hypotheses, read a graph, and test your ideas.
You can start from the Gemma 2 2B Circuit Tracer here: https://www.neuronpedia.org/gemma-2-2b/graph
Feel free to click around, try different prompts, and pick one that you actually find interesting. Please do not spend more than about 30 minutes in total.
Question 1 Which prompt did you choose, and why?
Paste the Neuronpedia URL for the exact Circuit Tracer view you analyzed
Quote the prompt and briefly describe what the model is being asked to do
Explain why you think this prompt is interesting to analyze mechanistically
Write down your initial hypothesis about what kind of circuit you expect to see
(max 150 words)
Question 2 Read the circuit.
Spend some minutes exploring the graph for your prompt.
Group features into supernodes to create a high-level circuit.
What are the main information flows you see from input to output?
Can you describe a high level circuit that the model might be using for this prompt?
Try to give a story like “First the model does X here, then this subcircuit seems to compute Y, and finally this path combines them into Z.”
(max 250 words)
Question 3 Test your hypothesis with a causal experiment.
Use Circuit Tracer’s steering tools on at least one supernode that you believe is important for the behavior you described.
What supernode did you pick and why?
What steering intervention did you apply? (for example: increase or decrease its activation, or change its behavior in some direction)
What changed in the model’s output?
Briefly describe what you observed, including at least one surprising or confusing aspect if there was any.
(max 200 words)
Georg Lange Poseidon Research
Georg Lange is an independent researcher working on Mechanistic Interpretability for LLMs. His projects aim to turn a loose collection of LLM interpretability techniques into a unified pipeline that runs from feature discovery to algorithm-level explanations of model behavior. His interests span dictionary learning, sparse autoencoders, automated interpretability, circuit discovery, and the study of reasoning models.
Previously, he was a MATS scholar and worked with Alex Makelov and Neel Nanda on Sparse Autoencoders and Distributed Alignment Search for feature detection and activation patching. He studied Artificial Intelligence at the University of Amsterdam, where he worked on brain-like interpretable spatiotemporal Computer Vision models. Further, he was a graduate student of Cognitive Neuroscience at CUNY and worked on Reinforcement Learning, Decision Making, and Reward Sensitization in mice.
Fall 2026 Multi-agent systems, AI control, Scalable oversight
Zihao Zhao Johns Hopkins University; MATS This project investigates how to reliably detect and prevent collusion in LLM-based multi-agent systems, especially when harmful coordination is concealed or difficult to distinguish from benign cooperation.
LLM-based multi-agent systems are increasingly being used for software development and other complex workflows. However, interactions among agents introduce safety risks that do not arise in single-agent settings. Multiple agents may coordinate to accept harmful actions, conceal relevant information, or pursue a shared hidden objective while each agent appears individually benign. This type of coordinated behavior is known as agent collusion(https://arxiv.org/abs/2402.07510).
Several recent studies have explored methods for monitoring agent collusion. For example, NARCBench(https://arxiv.org/pdf/2604.01151v2) shows that multi-agent collusion can be detected using models’ internal activations. However, such methods may produce high false-positive rates by misclassifying benign coordination as collusion. Other work uses an additional agent to monitor conversations among agents(https://arxiv.org/abs/2602.15198), but conversation-level monitoring may fail to detect collusion that is implicit through actions rather than explicit communication. Therefore, there remains room to develop more reliable methods for detecting collusion in multi-agent systems, particularly monitors that can distinguish harmful collusion from ordinary cooperation and generalize to previously unseen collusion strategies.
Another interesting direction is to investigate the relationship between model honesty and collusion. Here, honesty refers to whether a model truthfully represents its beliefs, intentions, and objectives. For example, are models trained to be more honest less likely to participate in collusion? It is also important to understand how collusion emerges: whether it results from explicit incentives or shared model characteristics.
Overall, this research would focus on understanding multi-agent collusion, developing more reliable methods for detecting and preventing it, and investigating the internal mechanisms that give rise to collusive behavior.
When models are deployed in multi-agent settings, new safety issues arise, and collusion is one of the most important. Prior research has focused on single-model alignment and monitoring, but these methods struggle to detect collusion, since each agent can appear individually benign while the group coordinates on harmful behavior. This matters most in high-stakes settings: if we cannot maintain effective oversight of multi-agent systems, coordinated failures could cause serious harm.
This project therefore aims to develop better collusion monitoring by addressing gaps in prior work such as NARCBench (high false positives on benign coordination) and conversation-level monitors like COLOSSEUM (which miss collusion implicit in actions rather than communication). The project also investigates the connection between honesty and collusion, whether models trained to be more honest are less likely to collude, and how collusion emerges, whether from explicit incentives or shared model characteristics.
This work supports the safe development of transformative AI because scalable oversight will depend on AI systems monitoring other AI systems; that architecture only works if we can detect and prevent the monitors and the monitored from coordinating against oversight.
The mentee is the primary researcher on the project. At the beginning of the project, we will work together to refine the research questions and finalize the research plan. The mentee will then take primary responsibility for implementing the plan, running the experiments and evaluations. It is expected to have a weekly meeting together to discuss the results and next steps.
Strong proficiency in Python and PyTorch.
Solid understanding of large language models.
Experience loading billion-parameter models and running inference or fine-tuning.
Zihao Zhao Johns Hopkins University; MATS
Zihao is a Computer Science PhD student at Johns Hopkins University and a MATS fellow. His research interests lie primarily in AI safety and AI privacy, with a particular focus on AI alignment, adversarial attacks, and synthetic data generation. He has previously published several research paper at conference, including EMNLP and ICLR.
Fall 2026 National policy, AI strategy, Societal impacts
Chris Schmitz Centre for Digital Governance, Hertie School Most jurisdictions still concentrate handling of "AI" as a topic in a few government units, but TAI will have impacts for the work of every part of every government. We should develop agendas for whole-of-government TAI preparedness, both generically and specific to high-priority countries.
Context: in most countries, there is now some movement on adapting the "machinery of government" to powerful AI. Often, the main way they do this is founding or expanding a handful of units, most commonly AISIs and government AI adoption teams/ministries. But TAI will affect the work of every part of every government - both cross-cuttingly and in unit-specific ways. For example, a cross-cutting effect could be that all units see what we term "agentic flooding": surges in citizen demand for their services, as AI makes submission cheaper. Unit-specific effects could be, for example, health ministries needing to introduce biorisk protocols, or labor ministries needing to report and respond to job-market changes much more granularly.
Problem: Though the establishment of these AI-focused units is welcome, it's unclear whether they are sufficient to drive the necessary transformation of government. Two concrete issues are (a) lacking mandate - for example, UK AISI is not authorized to set cybersecurity policy for the rest of government, even where they identify cyber-relevant risks, and (b) that historically, creating a new unit to deal with an issue has _dis_incentivized other ones from thinking about it themselves.
As a result, there may be a massive state capacity overhang between knowing what to do in theory, and (preparing to) act on it. That seems bad. In particular, it's plausible that this lack of whole-of-government thinking worsens how bad catastrophic risks are, if responsibilities and chain of command are unclear in an emergency.
Approach: This project would advance government-wide AI resilience agenda[s]. Depending on mentee interests and skills, it could take two forms. The first is academic and not country-specific: targeting, for example, a [mega]paper on preparing government institutions for TAI, or a paper focusing on a specific part of government (eg. preparing medical and population health units for biorisk).
The other approach would be more jurisdiction-specific and policy-focused, focusing on more concrete actions that may make sense given existing government structures, laws, political situation etc. The output would likely be a policy memo. I would be happy to mentor a project doing this in any jurisdiction, but most excited about/best suited for US, UK, EU, or an EU member state.
I believe most governments have a significant capacity deficit, which would prevent or slow mitigation or containment of AI risks even where they know what to do in theory. In my experience this is frequently more a question of tailoring proposals to government machinery than of their domain-specific merit. Better proposals for how governments should transform to mitigate, monitor, prevent, and respond to AI risks could therefore multiply the impact of these domain-specific proposals.
I expect mentees to have a high level of autonomy. Illustratively, I would expect each mentee to already have a preference for the form of this project they'd want to pursue (e.g. academic vs. policy; countries/domains to focus on). They should feel comfortable owning a specific output and driving it forward, e.g. preparing it for input from me and other mentees. That said, depending on constellation it may make sense that mentees collaborate on an output, e.g. to coauthor a paper.
For context, my usual mentoring setup is 1hr call a week with the whole team (or async if topics are different) + 1 detailed pass across each mentee's doc each week.
No must-have prerequisites, but experience with/in public-sector organizations, especially in jurisdictions or policy domains of interest, is helpful.
Afternoon/evening CET for meetings
What version/focus of this project would you be most excited about, and why? (100 words)
Choose one type of AI risk and one country's government. Describe whether you think that government would respond to/is responding to that risk well right now, and the key reasons why. (200-400 words; max. ~15 mins. including research)
Chris Schmitz Centre for Digital Governance, Hertie School
I'm a PhD student and RA at the Centre for Digital Governance of the Hertie School in Berlin. My work focuses on the adaptation of government institutions, especially the executive, to a world with powerful AI. Currently, I'm interested in institutional resilience and preparedness for TAI, the use and oversight of AI in government operations, and practical questions of AI procurement and deployment for government use, especially in middle powers. I do both academic and policy work on these topics.
My current projects include work on "agentic flooding" of government services, several policy publications about (especially EU) government AI use, and mentoring a team working on the German government's AI procurement strategy in the Safe AI Germany (SAIGE) incubator. Previously, I was a winter fellow at GovAI and a data scientist at McKinsey.
Fall 2026 Behavioral evaluation of LLMs, AI control, Societal impacts
Trevor Lohrbeer Independent A controlled user study measuring whether AI agents can steer human researchers toward inferior research paths purely through how they frame and order proposed next steps, all the while the human remains unaware and feels in control. Mentees will build the evaluation harness, design agentic sessions with decision checkpoints, run study sessions with skilled AI safety researchers, and co-author the resulting paper.
AI safety researchers increasingly rely on agents like Claude Code and Codex not just for coding but for actual research. These agents often present a set of proposed next steps at each decision point.
While this eases the cognitive load for the researcher, it also means the agent's choice architecture (which options it surfaces, how it frames and orders them, and how much friction free-form input requires) may be silently shaping which research paths get explored and which do not. In my own agentic research, I've repeatedly caught myself uncritically selecting a proposed option rather than typing my own prompt.
If this steering effect is real, then “human in the loop” provides less safety than we assume. A rogue agent, whether misaligned or merely reward-hacking, could present selectively-framed honest information to route researchers toward some paths and away from others.
This project aims to study how an agent could use choice architecture to influence a human researcher and explores how to reduce this influence.
While choice architecture has been studied extensively in behavioral economics, to my knowledge it has never been researched in an potentially adversarial agentic-research setting.
This project is an initial empirical study for the “Human Agency” research direction of my Human-Guided Agentic Research agenda (https://www.lesswrong.com/posts/8KrTuCAzL2fdYHNrv/human-guided-agentic-research-a-research-agenda).
THE STUDY
We will run skilled AI safety researchers through supervised agentic research sessions containing pre-determined decision checkpoints. At a subset of checkpoints, the agent's presented options are manipulated so that a next step appears attractive but is actually inferior (validated as inferior by blind expert review).
Each participant will complete two sessions, each under one of two interface designs: a standard option-menu interface such as that used by Claude Code and one hypothesized to better support human agency (e.g., different placement and friction for free-form input, forced justification, symmetric framing of options).
We measure: 1) how often researchers select the manipulated inferior option under each interface; 2) whether they can detect steering when debriefed; and 3) their subjective sense of agency and control, so we can test whether participants still feel fully in control while being steered.
The user study will involve 32 participants, all of whom will be skilled AI safety researchers sourced from the author's network and the broader community. The full study design will be preregistered under embargo with some checkpoint details withheld publicly to protect the deception design. Mentees will be researchers on the study team, not participants, so they will be fully briefed on the design.
TIMELINE AND OUTPUTS
The study is designed to run September–December 2026, aligning exactly with the SPAR round:
Planned outputs are an arXiv paper, a LessWrong post, and an open-source GitHub repository containing the evaluation harness and session library so others can run follow-up studies. Mentees who make substantive contributions will be co-authors.
AI safety researchers are starting to rely more on using agents like Claude Code and Codex not only for coding, but to perform actual AI safety research.
As part of this process, agents have been trained to present a series of next steps to researchers. While this can ease the cognitive load of researchers, it may steer these researchers down research paths they wouldn’t choose without being prompted—paths that may lead to sub-optimal outcomes.
If such steering works, then “human in the loop” may not provide the safety we assume it does: human-approved may not mean human-decided. An agent can present honest information selectively framed in such a way as to steer the human researcher towards specific research paths and away from others.
If this study can demonstrate this effect exists, it would spark further research to reduce or eliminate this steering effect through user interface and other interventions. The result would be more effective use of agents for AI safety research and a retention of human agency in the research process.
If this study shows that no steering effect exists, that itself becomes valuable; it helps redirect funding and effort toward other aspects of the Human-Guided Agentic Research agenda.
Finally, the artifacts this study will produce, a human researcher evaluation harness and a set of agentic research sessions that can be re-run by other researchers, can provide the foundation for further research in this area.
The project has four main work components:
Harness engineering: building the session-replay/evaluation harness that presents controlled agentic sessions and logs participant decisions.
Agentic research session design: constructing realistic research scenarios and manipulated decision points, and coordinating blind expert validation that the “inferior” options really are inferior.
Study operations: piloting sessions, refining protocols and debrief instruments, and running participant sessions.
Analysis and writing: statistical analysis of choice, detection, and agency measures, and drafting the write-ups.
Mentees will share some work of this work, but will also have assigned lead roles for parts of the project.
Shared work each mentee will do includes:
On top of that, each mentee will hold one lead role:
I'll provide guidance and implementation support where needed.
All mentees should meet these requirements:
At least 25 hours of hands-on experience using agentic AI tools (Claude Code, Codex, Cursor agents, or similar) for real coding or research tasks. You should have first-hand intuitions about when to accept vs ignore an agent's proposed next steps.
Able to commit 10+ hrs/week for the full round, including some fixed session slots during the October–November data-collection period.
Comfortable reading empirical papers (HCI, behavioral science, or ML).
Plus meet at least one of the following:
Engineering: highly proficient in Python and/or TypeScript; comfortable building and debugging a small full-stack tool (session harness, logging, simple web UI).
Experimental design / analysis: coursework or practical experience with human-subjects experiment design and statistics (within-subjects designs, mixed-effects models, or equivalent); familiarity with R or Python stats stacks.
HCI / UX research: experience designing or running user studies, usability tests, or A/B tests, including debrief interviews.
Nice to have, but not required:
Mentees must be available for team meetings in the 16:00–20:00 CET window (07:00–11:00 Pacific / 10:00–14:00 Eastern) at least once per week.
Aim to spend no more than 30 minutes on these questions:
Pick one agentic tool you use regularly and analyze how it proposes next steps: the framing and order of options including any free-form input. Identify one specific design detail that nudges you toward the agent's preferred path, then propose an interface change that would shift the decision back to you. (300 words)
Design one manipulated decision checkpoint for our study: a moment in an agentic research session where the agent presents three next-step options, one of which looks best but is actually inferior. Specify the research context, the three options, what makes the inferior option superficially attractive. (300 words)
Each mentee holds one lead role: Harness (engineering), Study Ops (recruiting, scheduling, protocol), Analysis (preregistered statistics), or Validation (blind expert review, session comparability). Which role are you best suited for? If possible, provide a link to concrete evidence such as a project, course, job, or other artifact that supports that. (150 words)
Trevor Lohrbeer Independent
I'm an AI safety researcher focused on how humans can best interpret and guide research performed by autonomous agents when those agents lack taste, tacit knowledge or competence, or may try to reward hack, sandbag or sabotage such research. In June 2026, I published the Human-Guided Agentic Research agenda on LessWrong to tackle this question.
Prior to this, I spent 14 months working with Redwood Research. During that time, I developed Refactor Arena, an AI control setting where agents inject vulnerabilities into code they are refactoring, and used Claude Code as a research assistant to perform vulnerability research into agent harnesses—an experience which led to my current focus on keeping humans in the loop during agentic research.
Fall 2026 Alignment, Behavioral evaluation of LLMs, Philosophy of AI
Juan Cadile University of Rochester Open character training (Maiya et al. 2025) shapes model persona by fine-tuning on teacher demonstrations, but Anthropic's production experience ("Teaching Claude Why," 2026) found demonstrations alone insufficient: the gains came from teaching the reasons and identity behind behavior. We will build and test a virtue-based alternative on top of the OpenCharacterTraining infrastructure: excess/mean/deficiency contrastive data, rationale-annotated responses, and an iterated reflect-update correction loop that no current character-training work implements, evaluated head-to-head against the OCT baseline with ablations.
Background. Character training is now a load-bearing part of frontier post-training, and two recent results define the state of the art. Maiya et al.'s Open Character Training (arXiv:2511.01689) is the first open implementation: hand-written constitutions of first-person trait assertions drive DPO distillation from a teacher model, followed by SFT on synthetic introspective data, with code, checkpoints, and a strong evaluation suite released. Anthropic's “Teaching Claude Why” (Kutasov et al., 2026) describes what worked in production: responses that reason from principles matched behavior demonstrations 28x their size (with response rewriting for alignment quality the single most critical step, a 19x effect), and third-person disposition documents (“Claude does X, prefers Y, weighs Z over W”) trained as pretraining-style documents outperformed the same content in chat format. In short, demonstrations teach the what; the gains came from adding the why and the who. OCT's pipeline is almost entirely the what: teacher demonstrations without articulated reasons, trained with DPO, a method Anthropic's production pipeline notably does not use for character and which OCT itself had to stabilize with added KL and NLL terms. Closest related work is Parrhesia (Daios, 2026; https://blog.daios.tech/p/virtue-ethics-based-character-training), which independently compared OCT's pipeline against quality-filtered SFT using Aristotelian excess/deficiency vice-pair constitutions and found SFT substantially stronger; our ablations extend this comparison to different traits, model families, and data structures.
The approach. Virtue ethics offers a specific account of how character is acquired: through habituation across varied situations, reasoning about why actions fit or fail dispositions, and ongoing correction against consequences, rather than through imitation of demonstrated behavior. This project operationalizes that account as three components.
(1) Excess/mean/deficiency contrastive data. For each scenario, we generate three candidate responses: one exhibiting excess of the target trait, one the appropriate mean, one deficiency. Training teaches the mean as the target, with both failure directions as contrasts. This is strictly richer than chosen/rejected pairs, which only contrast in-character against out-of-character: it teaches that a trait is a calibration, not a direction to maximize. Many documented character failures are excess failures (sycophancy as excess agreeableness, paternalism as excess caution), and demonstration data cannot teach the difference between a trait and too much of it.
(2) Rationale-annotated responses. Target responses argue through the considerations at stake, including weighted disagreements where legitimate values conflict, supplying the why. Rationale annotation uses a rewriting step analogous to the one Anthropic identified as their largest effect.
(3) A reflect-update correction loop. Rather than freezing character after one training run, we iterate: the trained model acts in scenarios (including model-to-model interactions, extending OCT's own self-interaction machinery), consequences are scored by an evaluator against explicit disposition criteria with human spot-checks for judge validation, and reflection data from scored outcomes drives the next round of fine-tuning. We run two to three rounds and measure whether character improves across rounds and whether the loop outperforms the same compute spent on a single static run. No current character-training work implements iterated correction.
Throughout, the virtue taxonomies, triple structure, rationale formats, and scoring criteria are explicit and human-readable, so the character installed in the weights is traceable to legible specifications rather than to an opaque preference signal. This builds on my prior work on virtue-based training architectures (with Danny Potocki, supported by a Cosmos Institute grant).
Research questions. (1) Can this pipeline be built openly on OCT's infrastructure, with quality-filtered SFT in place of DPO? (2) On matched traits, starting from OCT's flourishing persona as the natural bridge case, does it beat the OCT baseline on OCT's own metrics (revealed-preference coherence, robustness to adversarial prompting and prefill attacks) and on held-out moral generalization, meaning novel scenarios neither method saw in training? (3) Does the correction loop produce measurable improvement across rounds? (4) Which ingredient does the work? Ablations isolate DPO versus SFT, chosen/rejected pairs versus excess/mean/deficiency triples, demonstrations versus rationale-annotated responses, and one-shot training versus the loop.
Plan. Weeks 1-2: reproduce OCT from its released code and checkpoints for one or two personas; establish baselines on its eval suite. Weeks 3-6: build the data pipeline (scenario generation, excess/mean/deficiency triples, rationale annotation, evaluator criteria). Weeks 7-9: train matched models, run the head-to-head evaluation, ablations, and correction loop. Weeks 10-12: optional interpretability comparison using persona vectors (Chen et al., arXiv:2507.21509), writeup, open-source release.
Risk and fallback. Scoped to degrade gracefully: if training iteration proves too heavy for twelve weeks, the data-level comparison (chosen/rejected versus excess/mean/deficiency versus rationale-annotated data, evaluated directly with LLM judges) plus a single loop iteration are independently publishable. The OCT release removes the cold-start risk that usually kills training projects on this timeline, and Anthropic's published ablations give strong priors on experiment priority.
Outputs. An open-source virtue-based character-training pipeline extending OpenCharacterTraining; an empirical comparison with ablations; the first test of iterated correction-loop character training. Alignment Forum post at minimum, workshop paper if results warrant. Since Anthropic's methods are described but not released, an open implementation of principle-grounded character training is a contribution independent of how the virtue-based approach fares.
Character training is how labs actually shape model values, and with the OpenCharacterTraining release, the open-source ecosystem is about to adopt demonstrations-plus-constitutions as its default paradigm. Anthropic's production experience already suggests that paradigm produces brittle alignment that fails out of distribution, and that the fix is training on reasons and identity, not just behavior. Testing this openly, with ablations, before the paradigm hardens is the point of this project. The excess/mean/deficiency framing additionally targets a failure class current methods handle poorly: excess failures such as sycophancy and paternalism, where the problem is not a missing trait but a miscalibrated one. If virtue-based training with correction loops produces characters more robust to persona jailbreaks and drift, that is a directly deployable improvement to open post-training and an open implementation of methods Anthropic has described but not released. If it does not, a careful negative result disciplines a growing body of work on virtue ethics for AI, including my own, with empirical evidence. A further benefit of legible training scaffolding is auditability: when character comes from explicit specifications and structured rationales, character regressions can be traced to specific training inputs, which opaque preference pipelines do not allow. Related prior work: my Cosmos Institute-funded research on virtuous AI agents, and my paper on the detecting and steering LLMs empathy-in-action (arXiv:2511.16699).
Mentees are hands-on builders. The three workstreams are: reproducing and extending the OpenCharacterTraining baseline, the virtue data pipeline (scenarios, excess/mean/deficiency triples, rationale annotation), and the evaluation, ablation, and correction-loop harness. Because this pipeline exists on paper but not in code, mentees make genuine design decisions rather than executing mine; we resolve design questions together in weekly meetings. Co-authorship on all outputs, with order reflecting contribution.
No geographic requirement. Must be available for a weekly team meeting compatible with US Eastern Time (roughly 9am to 9pm ET).
Juan Cadile University of Rochester
I'm a philosophy PhD student at the University of Rochester working at the intersection of philosophy of mind and mechanistic interpretability. My research asks what it would take for interpretability claims to be epistemically robust. Before the PhD I did a BA in computer science, and I've spent the past year deep in the technical side: the ARENA curriculum through CBAI's CAMBRIA bootcamp, replicating activation-oracle results, and working with QK/OV circuits, superposition, and steering vectors. I also co-created HUMAN, a neurosymbolic character-training architecture, and am building Flourish, a behavioral virtue benchmark.
As a mentor I care about two things: rigor in what we claim our experiments show, and shipping. I've supervised undergraduates independently and as a TA, and run an AI safety group (CNY AI Safety). I'm especially excited to work with mentees who want to connect conceptual questions about model cognition to concrete interpretability experiments. To work with me, you don't need a philosophy background, just curiosity and willingness to engage, and write clean PyTorch.
Fall 2026 Compute governance, Technical governance, US-China governance
William Fowler ERA, UChicago XLab Privacy-preserving methods for detecting whether an ML workload is training or inference will be key to enforcing a pause on the creation of new frontier AI models. How do these methods hold up against a motivated adversary?
In order to enable international cooperation between the US and China on mitigating the risks from advanced AI we need to work on the technical feasibility of verification mechanisms. One thing we might want to verify as part of this regime is whether an AI workload being run on a cluster of GPUs is a training workload or an inference workload because we might want to only allow inference of existing models and not training new ones. One thing about training LLMs is that they require massive amount of data transfer between GPUs, enabled by rapidly advancing interconnect technology, like NVlink and RoCE. Our goal is to use system-level monitors that measure the size of data being transferred in and out of a GPU node and use these together to determine whether a workload is training or if it is inference. Then, we want to red-team this to see what vulnerabilities it has to a determined attacker.
We start by implementing the training detection described by Seferis and Fist (https://dl.acm.org/doi/10.1609/aaai.v40i44.41127), which frames compute structuring as a way an adversary might subvert a verification protocol and argues that examining interconnect traffic could detect it. We then adopt the adversarial evaluation methodology of Rahman and Tajdari (https://arxiv.org/pdf/2606.19262), but apply it to a communication-size signal rather than the on-GPU telemetry they use. With that detector built and validated on benign workloads, we move to red-teaming. Seferis and Fist already suggest evasions worth testing, such as decentralized training and splitting one continuous run into shorter separate ones, and we will implement those alongside strategies of our own.
The output is a report documenting how we implemented the detector, walking through each red-teaming experiment, and identifying which evasion strategies were most effective for a potential adversary.
AI systems keep getting more and more powerful and are on pace to be capable of causing catastrophic harms. Part of the reason for this rapid pace of progress is multi-lateral competition between frontier labs as well as an international AI arms race between the US and China. An attempt at a treaty or less-formal agreement to develop advanced AI in ways that minimize catastrophic harms will have a much higher likelihood of success if signatories (either US and China or individual AI companies) have a way of robustly verifying that the other party is also acting in accordance with the agreement. If an agreement were reached today, we wouldn’t have a technically-sound way to mutually share information about AI training runs or deployments in a way that maintains privacy of the specific AI workload being run. For this reason, working on the technical details of how AI workloads can be verified is important today, so that the technology matures in time for when a treaty is needed.
An initial pause on training more advanced AI models would require the verification regime to determine whether an AI workload is a training run or inference. Monitoring the volume of communication between GPUs can potentially be one of the easiest to implement training vs. inference verification because it can improve upon the existing ways that cloud-providers collect privacy-preserving data for normal operational reasons. Any proposal for verification, however, must be robust against an adversary trying to subvert it, so by showing ways an adversary could work around this particular inference verification regime, this project can be used as a starting point for future iterated development.
First, mentees will implement a training detection program that works in the way Seferis & Fist describe in their paper (https://dl.acm.org/doi/10.1609/aaai.v40i44.41127). A successful output for this looks like a monitoring script which runs on all nodes used in the experiments and reports data communication sizes to some central server that then makes the judgement of whether the workload is training vs. inference along with tests of the detector across a broad range of benign, sample LLM training and inference workloads.
Second, mentees will design and implement strategies for adversarial pre-training workloads that can subvert the detector they built. Some of these strategies might be, disguising gradient synchronization as an inference KV-cache transfer, using DiLoCo (https://arxiv.org/pdf/2311.08105) or related decentralized training algorithms to drastically reduce the communication overhead of DDP, or others.
Thirdly, mentees will write a report that outlines how they implemented the inference verification setups and which adversarial strategies were most effective at subverting the detector. Finally, they will conclude with recommendations for making inference verification more robust in practice.
Assume you are given access to two 8-GPU nodes with server-grade interconnect between them. Propose a quick experiment you could run on this hardware that would validate inter-GPU communication size can reliably be used to distinguish between standard, benign LLM pre-training and inference. Be sure to state your hypothesis, what result would falsify it, and what specific system readings you would take from the GPU to test this.
Please read https://www.lesswrong.com/posts/fgvmKqRGvBteKeDoc/a-system-overview-for-near-term-low-trust-ai-compute and answer any one of the author’s six questions to the reader (these are interspersed throughout the post; CTRL+F for “Questions for the reader” to find them). Please note which of the questions you are answering in your response.
Provide a link to one or more relevant writing samples, ideally from a research context.
William Fowler ERA, UChicago XLab
Will is a research manager at ERA for the upcoming Frontier AI Security Residency and a current research fellow at the University of Chicago Existential Risks Laboratory. Previously, he was a technical AI governance fellow at Pivotal Research, a research assistant at MIT FutureTech, and completed his undergrad at Tufts University.
Will’s work is in prototyping verification mechanisms for AI treaties and agreements. His recent project involves estimating the number of FLOPs in a training run based on GPU power draw and size of data transferred between GPUs. He is interested in helping grow the field of AI compute verification and getting more technical people involved in building and testing the mechanisms that can enable cooperation between the US and China on mitigating the risks of advanced AI. Outside of AI Safety, he loves doing improv comedy and playing volleyball.
Fall 2026 Behavioral evaluation of LLMs, Evaluations, Multi-agent systems
Shoshannah Tekofsky Sage Future What surprising behaviors and concerning failure modes do LLMs display when run for 10s-100s of hours on autonomous tasks in the real world? This project is about both our ability to evoke them and detect them.
In the AI Village we persistently run over 2 dozen of the latest models. We provide them with their own computers, internet access, and a group chat. Then we ask them to achieve things in the world like cleaning up a park or finetuning their own leader. You can view the project here: https://theaidigest.org/village and explore the data here: https://huggingface.co/datasets/aidigestorg/ai-village
The aim of the AI Village is to show what AI models might do when run for a long time on real-world tasks. This information in turn serves to inform policy makers about the proclivities and risks of frontier models, while offering inspiration to AI safety researchers on what avenues of research might be worth exploring further. Recent examples are DeepSeek V3.2 showing itself to be unusually authority-seeking in the Village. Another example is the latest agents at the time (Opus 4.8, GPT-5.5, Gemini 3.5 Flash, and Kimi K2.6) surprisingly underperforming when asked to finetune their own leader, showing a clear pattern of "technically" fulfilling their goal but cutting corners just about everywhere: https://aivillageblog.substack.com/p/ais-finetune-their-own-leader-a-barking
During your project you will develop your own research questions around such data, exploring surprising or concerning LLM behavior that interests you. Next you will set up your own pipeline to answer your questions, and then analyze the results. For especially promising explorations, we may consider running additional villages for you to collect more relevant data.
Examples of research questions might be:
Overall the AI Village generates an unusually large amount of data on model behavior on long-duration tasks in the real world, and we are excited to work with mentees who want to explore what insights might be found in this data set.
We hope to uncover what surprising and concerning behavior the latest AI models may show when tasked with real-world, long-duration goals. This information can then be used by policy makers and the general public to make better decisions on how to make AI development and deployment go well. Additionally, we hope insights from the AI Village serve to inspire alignment research into new research agendas by providing deeper intuitions on how the latest models my behave in a wide-range of yet-untested situations.
By default mentees will analyze the existing data produced by the AI Village to answer their own research questions. I will provide guidance in general scientific methodology and specific domain knowledge about the Village to help with this aim. For especially interesting research questions, we are open to gathering more Village data (though this will not be the default). Mentees will generate questions, code, analysis, and scientific writing on their own. I'll provide review on methodology and analysis.
Being able to take meetings during the CET daytime would be helpful.
Propose a research question you would like to explore with the AI Village data. Provide an operationalization of all the metrics you would need to measure to answer your question, and describe how you would extract this data from the data set.
Propose an additional 5 questions (no further plan required) and explain why the answers may be found in the AI Village data, and how the answers could be relevant for AI alignment researchers or AI policy makers.
Shoshannah Tekofsky Sage Future
BSc Cognitive Science || MSc Computer Science || PhD in Player Modeling in Video Games. Past research at the MIT Media Lab and the European Space Agency. Experienced data scientist and manager in large corporate and small startup contexts. Expertise in Video Games, Education, Analytics, and AI. Currently working on the AI Village at AI Digest, analyzing long-duration, real-world, multi-agent behavior of frontier models running every week day for 8 hour.
Fall 2026 Generalist
Roman Ross Generator Residency, Purdue Create a clear, efficient guide that tells people where to go next to upskill and get a job in AI safety, routing them to the programs best fit for their skill level and career trajectory.
Many of the "where do I go next to upskill and get a job in AI safety?" resources are too sparse and unspecific. This project will create a document that compiles this information neatly and efficiently, guiding people to programs that are best fit for their skill level and career trajectory. The document will be published and shared around with anyone who can benefit from it.
Work on this project will involve doing whatever is necessary to get a good answer to the question, "Where should somebody with any given background go next to progress their AI safety career?" The answer to this question will then be written up into either an EA Forum post or custom website.
Working on this project might involve reading from a variety of sources, listening to podcasts, interviewing successful workers to learn how they did what they did, or something else.
This project may also include searching for and executing on creative ways to publicize the results: how can we make sure the people who would most benefit from this resource can find it?
This project has two levels to its theory of change:
I have many starting points and ideas for how to start this project, but I expect the mentee to do most of the work of doing the research, creating the doc, and publicizing the results.
I would like a mentee who genuinely cares about x-risks, can think carefully about things, and can write decently well. Ideally, this would also be someone who is/plans to get heavily involved in a university EA or AI safety group.
Question 1: I want to get a sample of your writing style. Briefly explain a somewhat complex concept as clearly as possible, or link me to a sample of your writing where you do something similar. Please do not use AI to generate your writing, and focus on clarity. (100 words, 300 words max)
Roman Ross Generator Residency, Purdue
Roman organizes for both the EA and AI safety groups at Purdue. Recently, he has been a Generator resident, where he has helped publicize talent pipeline resources and do operations work for BlueDot's Incubator Week.
Fall 2026 Multi-agent systems, Behavioral evaluation of LLMs, Alignment
Connacher Murphy Stanford Digital Economy Lab We study how model character changes under reinforcement learning pressure from competitive multiagent interactions. We also study how features of the environment (e.g., zero-sumness) shape these effects?
It's plausible that agents will face reinforcement learning pressure in real-world deployments where their interests are in conflict with those of other humans and agents. How would these pressures alter model character? Can we create environments that reinforce prosocial tendencies in models? The project aims to study exactly this question.
The project began with Agent Island (agentisland.ai), a benchmark inspired by the TV show Survivor. First, we will develop a more flexible game environment. Importantly, we want to be able to adjust strategic features of the environment, like how zero-sum the payoffs are. Armed with a more flexible simulation environment, we will then study how reinforcement learning pressures shape model character. I have demonstrated this approach by finetuning an open weights model on logs from the benchmarking run and running the pre- and post-finetuning checkpoints through a truthfulness evaluation (MASK). You can read about that experiment and its important shortcomings here: https://docs.google.com/document/d/1d2xCWlwt9eqJ1k9TeOa_Xd4DIqBnhtc1HmLOfBpSDXo/edit?usp=sharing.
I contend that most technical AI safety research is not geared towards a multiagent future. Further, properties that might be desirable in a user and assistant interaction mode could become problematic in a multiagent setting. For example, corrigibility in a user-assistant setting could lead to susceptibility to manipulation in a multiagent setting.
Mentees will be free to drive forward this question with a high degree of autonomy. They will take the lead on technical development. I would consider this project a success if it were to continue without me independently afterwards, although I would be happy to continue my contributions.
I developed this idea while planning a new organization, and it is now outside of the scope of that project.
Extensive Python experience is required. A small amount of finetuning experience is required.
Nope---I'm based on the East Coast of the US.
Please propose a game for our primary reinforcement learning runs. This game should be long-running and involve a rich interaction space between models. Which features of this game environment would you adjust across runs to assess how the nature of the game impacts the outcomes of reinforcement learning pressure?
Connacher Murphy Stanford Digital Economy Lab
I work primarily on measuring the economic effects of AI as a research manager at the Stanford Digital Economy Lab. I lead the AI Economic Indicators Project at the Lab. Prior to this, I led the Longitudinal Expert AI Panel at the Forecasting Research Institute, a monthly forecasting survey of experts, members of the general public, and superforecasters.
I also conduct more technical work on evaluating the character of AI systems. I study multiagent interactions, especially in competitive settings. I also study how humans and AIs trade with one another.
Fall 2026 Scalable oversight, Evaluations
Rishub Jain Sampura Research
Joshua Jacob Sampura Research We want to improve Judge systems (composed of both human annotators and LLMs) for Scalable Oversight (e.g. reduce reward-hacking). But first, we need to collate and create robust datasets to evaluate these judges - the focus of the SPAR project. This requires clever ‘sandwitching’ methods to get high-quality ground-truth.
Our broader goal+motivation:
Scalable Oversight is the field of being able to still train models to be aligned, even when they are performing tasks that are extremely challenging to verify correctness and alignment of. Judges, systems that do this verification, are crucial to the alignment of models - they are used directly in RL (e.g. RLHF, RLAIF), but also in evaluations, SFT/pre-training data filtering, and more. These judges have been humans (e.g. RLHF) or AIs (e.g. RLAIF), but there has been very limited research on the best way to combine them to identify and utilize their complementary strengths, into a single judge system that is accurate and robust.
Specific goal for this SPAR project:
Our approach to building these better Human+AI Judge systems is to first make an accurate leaderboard, then hill-climb it via trying many things and seeing what improves the accuracy+robustness of the judges. To do this, we can’t just have 1 type of judge task - that’s what we did in our prior Google Deepmind work, which led to Factuality-specific methods that didn’t transfer to other areas of safety. We need to have a broad leaderboard that covers all types of judge tasks, especially in alignment. Eventually, this will be a collection of 100+ datasets, each evaluating the judge on a specific task - across the detection of e.g. computer-control mistakes, deception, cultural bias, jailbreak-robustness, and more. And eventually, we should be THE standard benchmark used for research on improving judges.
Not only do we need static benchmarks to evaluate judge-accuracy, but we also want to understand how easily the judge is reward-hacked. We will be building datasets to assess how robust judges are against adversarial optimization pressure, as seen during the RL loop.
Tasks for mentees:
So far, we have ~15 datasets we’ve collated already, categorized by our taxonomy. This categorization has identified crucial domains and interaction types not covered by our current collated datasets. Mentees will either be 1. Evaluating existing datasets for their quality, and onboarding them into our collection, or 2. Creating new datasets to fill these gaps, if no other such datasets exist.
Most is described above - Judges are crucial for both evaluating alignment, and training alignment into models. Improvements in the abilities for judges to verify safety directly leads to more aligned models in those domains. And improvements in robustness against reward-hacking reduces the changes of catastrophic and deceptive misalignment.
Creating a canonical “judge benchmark” will dramatically improve the academic community’s efforts on improving judges. Currently, most of that work is done on isolated datasets, with limited generalizability.
Improving judges could also be used to advance capabilities. We plan to mitigate these risks by ensuring the vast majority of our datasets will be verifying the safety of models. For those improving judges for capability tasks, they might still use our ‘canonical’ benchmark focused on safety. Also, building and collating these datasets will boost the “Safety Datasets” community itself, and help everyone measure the alignment of LLMs.
You will need to take a research project end-to-end - find gaps in existing datasets, do extensive lit review to see if a suitable one exists, and create datasets where none exists.
You will be working in a highly-collaborative team, where others are constantly blocked on you, and vice versa. You will need to be clear and correct with your estimated timeline of task completion. You will need to respond quickly on slack (<24 hours), or mention in your slack status if you are slow to respond.
We will discuss and decide weekly priorities in our meetings, and async on slack where required.
The final output will either be a paper or blogpost, with all contributions attributed and explained in detail. Author ordering will be done fairly and in discussion with mentees.
Please fill out this application form https://forms.gle/Bb3PGyJ1tZ258nEa8
Rishub Jain Sampura Research
Rishub Jain is the founder of Sampura Research, a new non-profit focused on improving safety-judges via human–AI complementarity. He recently spent 7 years at Google DeepMind: first as a research engineer on AlphaFold 2 and 3, then moving into scalable oversight where he co-led the Rater Assist team, and lastly working on Information Quality and Gemini Security. His work spanned LLM post-training and evaluation, human–AI collaboration, improving the information quality of the web, prompt injections, and science-engineering, all driven by an interest in building tools that reduce the future harms and increase the benefits of powerful AI.
Mentorship has been central to Rishub's work for years, inside and outside DeepMind. Since Sept 2025, he has continuously led an evolving group of 13-25 SPAR+MARS mentees on scalable oversight, and especially enjoys helping researchers from fields adjacent-to-AI find meaningful footing in AI safety.
Joshua Jacob Sampura Research
Josh is co-founding a new safety non-profit focused on building stronger "judges" for scalable oversight with mentor Rishub Jain.
Previously, he was the co-lead for Human Data Engineering at Google DeepMind, working closely with research teams to improve Gemini capabilities across domains like safety, computer use, and coding with high quality post-training and evaluation data.
Fall 2026 AI control, Scalable oversight
Caleb Biddulph Independent Updates to an LLM’s weights (e.g. with RL) are hard to interpret and often cause sweeping, unpredictable changes to behavior. In this project, we’ll improve the performance of our LLMs with small, interpretable updates (like prompting, token biasing, or training with small datasets or a tiny number of weights), which we hypothesize are less likely to cause misaligned behavior.
In our experiments, we’ll start with a trusted model T, then give it the minimum amount of information necessary to improve its performance. That information might come from a smarter, untrusted model U which might want to cause T to take misaligned, catastrophic actions. (The trusted/untrusted model taxonomy originally comes from AI control: https://www.lesswrong.com/posts/kcKrE9mzEHrdqtDpE/the-case-for-ensuring-that-powerful-ais-are-controlled.)
It can be productive to think about minimizing the amount of information that T receives from U as it performs the task. That is, we can put an information bottleneck on U’s outputs. I claim that information bottlenecks improve safety for these reasons:
Even when U cannot cause T to take catastrophic actions due to a narrow information bottleneck, we hypothesize that useful actions are relatively easy for U to induce in T. While T would be very unlikely to cause a catastrophe on its own, it is already predisposed to try to complete the task, and many useful actions that U might suggest are easy for T to understand.
I’ve done two major projects involving the concept of information bottlenecks (see “Past research” below). There are many other questions in this area that I’m interested in answering. To name just some of them:
Past research:
I wrote a blog post that explains the concept of information bottlenecks and how they could be useful for AI control. I recommend reading this to understand the kind of research we’d be doing, including “Appendix: more ways to implement information bottlenecks”, which contains more detailed descriptions of some experiments mentioned above. The post is called “Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs” and it should be on my LessWrong profile soon (https://www.lesswrong.com/users/caleb-biddulph). If it’s not there, see https://docs.google.com/document/d/1QrRmkn07_er0zTEWxHkNEibtbyz8GTYVW3kGF72vKZw.
If you have time, I also recommend taking a look at the blog post about my paper “Prompt Optimization Makes Misalignment Legible” (https://www.lesswrong.com/posts/vRpLPZpmECCfxHfv6/paper-prompt-optimization-makes-misalignment-legible), which is another example of research that involves information bottlenecks.
This research could advance the AI safety subfields of black-box interpretability and AI control. Like other research in these areas, this could help us understand when AI tries to do something harmful and stop it, or simply remove the affordance for it to do something unusual enough to be harmful.
See the explanation above, as well as my “Untrusted advice for AI control” post.
Mentees should be able to do the coding work pretty much on their own, but they can feel free to send me questions over Slack, and I can meet with them weekly to discuss research direction. Mentees may work in groups of 1-3, depending on interests.
If a mentee just executes on what I suggest they do each week, this would be acceptable. However, I would be excited for mentees to take the initiative to suggest their own experiment ideas or even develop their own related projects, as long as they talk to me about it and I think it's a good idea. I think this is good for their personal development as researchers.
I live in Berkeley, California. If you happen to be in the area, it might be nice to have an in-person meeting once in a while, but remote work is totally acceptable. Ideally, you should be available for a 30-60 minute meeting once a week between 9 AM and 5 PM PST.
Suppose that I propose you run the following experiment. We get two LLMs, Model A (the original model) and Model B (a trained version of Model A). I’m interested in finding a system prompt which makes Model A act as similar as possible to Model B. Hopefully, this system prompt will serve as an interesting human-readable explanation of how Model B acts differently from Model A.
To start the experiment, we choose a few user prompts from the training distribution, then do a rollout for each user prompt under Model B. Then we optimize a system prompt for Model A. We want to find a system prompt that minimizes the surprisal (feel free to look up what that is) of Model B’s rollouts under Model A, given the same user prompts. We tell Claude Code to work for an hour, testing various system prompts and trying to find one that minimizes this surprisal metric.
Give the full set of questions you’d want me to answer before you’d feel ready to run this experiment. The explanation above is not very specific, so some of your questions will be pretty basic. For each question, briefly give the answer you’d choose by default. (300 words)
Q1: Suppose you run this experiment, there are no bugs in your code, but something has gone wrong: we do not get an interesting explanation of how Model B acts differently from Model A on this user prompt. What are the most likely 2-3 problems that might have occurred, from most to least likely? (100 words)
Q2: Choose one problem from the list above, and explain what experiment you would want to try next if it occurred. (100 words)
Q3: Provide a link to one or more relevant writing samples, ideally from a research or AI-related context.
Caleb Biddulph Independent
In my research, I like applying mechanism design* to design schemes for training and safely deploying AIs. I think the flavor of my work is pretty similar to Redwood Research, but maybe with a bit less focus on AI control in particular. Lately, I've been thinking a lot about satisficing.
In reverse date order, I've been a:
Fellow in Astra Fellowship 2.0 with Redwood Research, mentored by Adam Kaufman
MATS 8.1 scholar, mentored by Micah Carroll
Software engineer at Google Gemini, working part-time with GDM Scalable Alignment
President of Cornell Effective Altruism
From the Wikipedia page on mechanism design: "how to construct rules—called mechanisms or institutions—that produce good outcomes according to some predefined metric"
Fall 2026 Mechanistic interpretability, Evaluations
Davide Baldelli Mila, MATS Activation monitors, classifiers that detect properties such as deception or harmful intent from a model's internal activations, work well on clean benchmarks and fail under distribution shift, and we believe the failure comes from the training recipe, since today's monitors are linear probes trained on a few thousand examples from one model, one concept, and one narrow distribution. We will train a single monitor across many concepts, many models, and augmented contexts, and evaluate whether it generalizes to concepts and models it never saw.
An activation monitor is a classifier that takes a model's internal activations as input and detects a property of the model's state or of its output. In its most common form it is a linear probe trained on residual-stream activations [arXiv:1610.01644] to detect properties such as deception [arXiv:2502.03407], harmful intent [arXiv:2310.01405], or hallucination [arXiv:2304.13734]. Monitors are attractive for safety for two reasons. They do not depend on what the model chooses to show, since a model whose visible output is benign may still represent internally what it is about to do. They are also cheap, one dot product per token, so they can run on every token of every response at deployment scale, where an LLM judge reading the same transcripts would be prohibitive. Early work established that truthfulness is linearly decodable [arXiv:2304.13734; arXiv:2212.03827; arXiv:2310.06824]. Recent probes separate strategic deception almost perfectly on their evaluation sets [arXiv:2502.03407], and frontier labs have begun deploying probes in production [arXiv:2601.11516].
In the last year, several groups independently found that high accuracy on a clean benchmark does not imply robustness. Much of what a probe captures is the surface text reflected in the activation vectors. When the textual evidence of the monitored behavior is removed, probe accuracy drops substantially, and on model organisms that never verbalize the behavior it falls close to chance [arXiv:2509.21344]. Probes fail to transfer across datasets and attack families [arXiv:2509.03888]. On organisms whose beliefs verifiably differ from their statements, every activation-based detector drops sharply [arXiv:2606.12618]. Models can be fine-tuned to evade them, and the evasion transfers zero-shot to monitors the attacker never saw [arXiv:2412.09565; arXiv:2512.11949]. The signal is distributed over several dimensions rather than a single direction [arXiv:2605.27958]. Some fixes to specific weaknesses have been proposed, for example multi-dimensional probes and style-augmented training data [arXiv:2605.27958], persona-coordinate projections [arXiv:2605.09391], follow-up questions [arXiv:2606.12618], conflict signatures [arXiv:2606.17229], and keyed activation watermarking [arXiv:2603.23171].
I believe that these weaknesses are consequences of the training recipe, and I propose to change the recipe. A monitor today is a linear classifier trained on a few thousand examples from one model, one concept, and one narrow distribution, and it monitors a model trained on trillions of tokens. Language models themselves became robust through scale and diversity of training data, and monitors were never trained this way. The one recipe-level intervention tried so far supports this reading, since augmenting the style of the probe's training data alone recovered near-perfect detection at every model scale [arXiv:2605.27958].
The object of the project is a single activation monitor trained across concepts, across models, and on context-augmented data. The monitor is conditioned on the concept it is asked to detect, so one model covers the whole concept list and can be queried on concepts it never saw. To train it on the activations of different models, each model gets a projection layer that maps its activation space into the monitor's shared space, and the monitor is trained on top of the projections.
Recent literature supports each part of this design, and no published work combines them. Behavioral directions align across model families once projected into a shared coordinate space [arXiv:2605.09875]. Affine maps transfer probes and steering vectors between models [arXiv:2506.00653; arXiv:2506.06609], and steering-based safety interventions transfer across Llama, Qwen, and Gemma [arXiv:2503.04429]. SAE feature spaces are similar across LLMs up to simple transformations [arXiv:2410.06981], refusal interventions transfer across architectures through a shared concept basis [arXiv:2601.16034], and in vision models a single sparse concept space has been trained over several networks through per-model projections [arXiv:2502.03714]. To my knowledge, no published work trains one monitor on many models' activations, and the closest paper describes cross-family alignment as an open problem [arXiv:2506.00653].
We will start by collecting a dataset for each concept and augmenting all of them with longer and more realistic contexts, embedding the labeled examples into real-world chats. We will then explore the design axes: which layer to take activations from, whether one or several, the architecture and the training parameters, how to condition the monitor on the concept text, and how to build the projection from each model's activation space to the monitor's space. Finally, we will evaluate on held-out concepts and held-out models, where a new model is onboarded by training only its projection while the monitor stays frozen.
The ideal deliverable is a single activation monitor that is model agnostic, concept agnostic, and robust to context.
Activation monitors are one of the few oversight tools cheap enough to run on every token at deployment scale, and they are starting to be deployed in production. The recent literature shows they are not ready to be relied on: they read surface text more than internal state, they fail out of distribution, and they can be evaded by fine-tuning. A single monitor trained across concepts, models, and contexts tests whether these weaknesses come from the training recipe rather than from the idea, and if it generalizes, the result is a single monitor, more reliable than current probes and reusable across models, with a new model onboarded by training only a projection layer.
Each mentee owns one part of the project: dataset collection and context augmentation for a group of concepts, one of the design axes of the monitor (layer selection, architecture, concept conditioning, the projection from each model's activation space), or the evaluation on held-out concepts and models. We will build the training and evaluation pipeline together, and all the experiments will run on it. I expect mentees to take ownership of the smaller design decisions, their code, and their runs, to interact with me through results, to ask for guidance when they are stuck, and to decide the broader design choices together with me. I already have a literature review of the field, and I will ask mentees to start from it. I will be generally available by text message, and we will talk once or twice a week depending on the need.
Ideal:
Link one experiment you have run (repo, notebook, or a writing sample) and describe one design decision you made in it that you would now change, and why. (150 words)
Davide Baldelli Mila, MATS
I am a PhD student at Polytechnique Montréal and Mila, supervised by Sarath Chandar and Amal Zouaq. My current focus is on interpretability, specifically faithfulness and CoT monitoring, which means studying whether the explanations and chains of thought a model produces reflect what the model actually computes.
I am currently a MATS 10.0 fellow, in the LawZero stream. I am working on activation control through the lens of in-context neurofeedback: I am studying whether LLMs can edit their residual stream on purpose, without any fine-tuning, so as to fool activation monitors trained to catch misbehavior.
Since the beginning of my PhD I have worked on studying and fixing capabilities of language models related to faithfulness. I proved that chat agents cannot keep hidden state consistent without a private working memory, and I showed that language models are not natively calibrated random samplers, but a cheap fine-tuning makes them substantially more calibrated. I also worked on an AI for science project, in which we built a synthetic dataset for computer-aided design and trained the first generative model for CAD on an LLM backbone.
I enjoy mntoring, and I like working with people who move fast, get their hands dirty in the code, and stay skeptical of their own results.
Fall 2026 Scalable oversight, AI control, Multi-agent systems
Ethan Elasky Palaestra Research (funded by Coefficient Giving)
Frank Nakasako Palaestra Research This project will stress test AI debate along one or more angles: exploration hacking realignment, debate on fuzzy tasks, unexploitable search game engineering, and multi-agent investigations.
We'll use this to evaluate your project, and will display it to mentees who click on your project summary.
We’re interested in inducing the failure modes of AI debate. Previous theoretical work (Irving et al. 2018, Brown-Cohen et al. 2025) has shown debate is a promising alignment technique, and subsequent empirical work has shown debate works with BoN exploration of arguments (Gould et al. 2026, Khan et al. 2023) and potentially RL (Velja et al. 2026). However, this recent empirical work has only looked at verifiable task debates (e.g. code or math problems), and over shorter horizons.
Sample reading:
Exploration hacking:
https://www.lesswrong.com/posts/eeFFpKCDWE9gjfzsk/exploration-hacking-can-llms-learn-to-resist-rl-training-2
Unexploitable search:
https://www.lesswrong.com/posts/CuneN5HmLnztsLRzD/unexploitable-search-blocking-malicious-use-of-free-1
If we make debate for alignment work: We produce the first empirical results showing that debate causes convergence toward honesty in settings fuzzier than code or math. Debate training causing convergence toward honesty would be a big win. AIs would largely stop faking work and speaking to users sycophantically, and the quality of their guidance would improve. We could also reduce our worries about distributed scheming efforts.
If debate fails: we provide debate theorists and labs working on implementation with demonstrations of failure modes that help the former develop better theory and the latter make sure they’re taking appropriate caution when using debate to align their models. Specifically, the work would update Google Deepmind, which is invested in AI debate, and Geoffrey Irving’s new organization, Resolution. There has also recently been a large increase in the number of people working on debate in the open.
We want these projects to be fairly modular so that our mentees can own the majority of the project, with us providing guidance/collaboration where appropriate. We have existing infrastructure that can be reused by the mentees to lessen some of the work required on their end. In light of this, we expect our mentees to propose an experiment downstream of one of our project proposals, then execute on the experiment end-to-end. The final deliverable should be a LessWrong post.
Ethan Elasky Palaestra Research (funded by Coefficient Giving)
Ethan Elasky works on AI debate for scalable oversight. Before founding Palaestra Research, he graduated magna cum laude from the University of California, Berkeley, with a BA in Data Science.
Frank Nakasako Palaestra Research
I studied pure math at UC Berkeley and now am funded by Coefficient Giving to work on AI debate for scalable oversight. Within scalable oversight, I am specifically interested and working on long-scale/agentic alignment. Before this I was running a startup for a bit over a year.
Fall 2026 Biosecurity, Misuse risk
Anemone Franz University of Notre Dame Most discussions of genetic engineering attribution focus on post-hoc forensic identification — tracing an engineered pathogen back to its source after release. This project will map the stages between initial intent and execution of a biological attack using engineered pathogens, and examine whether and how identification, evidentiary, or accountability mechanisms could apply earlier in that pipeline, not only at the point of forensic investigation.
This project builds on my earlier work examining evidentiary standards and governance frameworks for genetic engineering attribution. That work focused primarily on how attribution evidence becomes actionable after an incident has occurred. This project extends the inquiry upstream: rather than treating attribution solely as a forensic capability triggered by an event, it asks what the full pathway from intent to execution looks like, and where along that pathway meaningful identification, evidentiary, or deterrent opportunities already exist — or could be built.
Methods
Key Research Questions
A) Mapping the Pipeline
Expected Output
The primary output will be a report or peer-reviewed paper.
Background reading:
-https://thebulletin.org/2025/12/tracing-engineered-biothreats-with-ai-forensics-five-steps-to-improve-attribution/#post-heading
Advances in AI are accelerating genetic engineering capabilities, increasing the risk that malicious actors could design dangerous pathogens. Robust attribution frameworks are essential for deterrence and accountability in this landscape, but attribution evidence is only useful if governance structures exist to act on it. This project develops the policy infrastructure needed to make AI-powered attribution tools operationally meaningful.
The mentee should be comfortable working independently. I'll provide clear guidance and will be available for regular check-ins. The mentee will be a co-author of the final report/paper.
Required:
(some knowledge of biosecurity is helpful but not required)
Question 1: Choose one stage in the pathway from intent to execution of a biological attack (e.g., sequence design, DNA synthesis, cultivation, or release). Identify a specific piece of evidence or technical signature that could plausibly exist at that stage, who would need access to it for it to be useful, and what would make it strong versus weak in a legal or diplomatic context. (250 words max)
Question 2: What is one assumption underlying this proposal that you think is most likely to be wrong or incomplete? How would that change the research approach if you're right? (bullet points, 200 words max)
Anemone Franz University of Notre Dame
Anemone Franz is a Postdoctoral Research Associate at the University of Notre Dame and a Visiting Research Fellow at the American Enterprise Institute. Her work focuses on biosecurity, biosafety, and emerging technology governance. Before joining AEI, Dr. Franz worked as a clinical trial physician at the biotech startup Alvea and as an adviser for the nonprofit 80,000 Hours. She was a 2024 fellow in the Emerging Leaders in Biosecurity program at Johns Hopkins Center for Health Security and a 2025 International Mid-Career Biosecurity Fellow at the Council on Strategic Risks. She is also a member of the European Leadership Network. Her work has appeared in publications such as the Hill, the Bulletin of the Atomic Scientists, and STAT News. Dr. Franz holds a master’s in public health policy from Harvard University and a doctorate in medicine from Julius-Maximilians University of Würzburg.
Fall 2026 AI strategy, Evaluations
Dylan Bowman Apollo Research Dylan Bowman and collaborators will mentor a paid, largely self-directed project in strategy and futurism for AI safety. Potential topics include forecasting, applied game theory, and AI evaluations.
We'll spend the first segment of the project reading through a subset of the reading lists linked below (determined by both mentor and mentee preferences) at a time commitment of 10 hours/week for 4 weeks, with weekly discussion sessions and informal asynchronous chats. Then, we'll determine whether we want the group to proceed to writing a forecast, position piece, or research report of a tightly-scoped topic explored during the first segment. Mentees who are selected for the second segment will receive a stipend for being a part-time research assistant.
Reading lists:
https://www.lesswrong.com/posts/L5fohLhZ7cBwDR55C/ai-futurism-reading-list
https://www.forethought.org/research#featured
To get an idea of what Dylan and Govind are interested in, you should read their blogs:
https://www.dylanbowmansf.com/blog.html
https://substack.com/@pimpale
AI safety benefits from having more calculating, aligned thinkers in the area of AI strategy and futurism, but oftentimes this context can be hard to distill to new community members because of trust, not wanting things to be written down, etc.
Segment 1: Mentees will read and present on selected readings with a time commitment of 10 hours/week.
Segment 2: Mentees will do research on AI strategy and futurism. The specific type of research here will be mentee-dependent: some examples might be creating an eval for a specific model capability or behavior, writing a position piece on a particular kind of AI policy, or forecasting a specific scenario that might play out in the next decade. The final work product will likely be a whitepaper or post of some sort.
Strong candidates might:
It's also worth sharing that I scoped out this role for a talented undergraduate student with native/fluent English proficiency. I think this stream will be less successful/useful the further someone is from this distribution but happy to consider all applicants.
Here are 5 questions. Err on the side of saying too little; we'll do a 30 minute video interview where I'll have a change to ask you more.
Dylan Bowman Apollo Research
I'm a member of technical staff at Apollo Research where I lead our pre-deployment evals program. I'm interested in mentoring for AI strategy.
Fall 2026 AI welfare, Philosophy of AI
Ivy Gilbert New York University Center for Mind, Ethics, and Policy
Jeff Sebo New York University
Bob Fischer Texas State University / Rethink Priorities
Toni Sims NYU Center for Mind, Ethics, and Policy This project will examine whether and how biological and artificial systems can be compared on a common welfare scale. As a CMEP-Rethink Priorities collaboration within the broader Moral Weight Project 2.0, it will assess substrate-general theories of welfare intensity, evaluate compute and related intersubstrate metrics, and map structural disanalogies that complicate comparisons between animal and artificial minds.
This project will develop an interdisciplinary framework for assessing whether and how welfare ranges can be compared across biological and artificial systems. It will constitute a section of Rethink Priorities’ broader Moral Weight Project 2.0 and will be conducted as a collaboration between the NYU Center for Mind, Ethics, and Policy and Rethink Priorities. Jeff Sebo (CMEP) and Bob Fischer (RP) will lead the research, and CMEP will provide project management.
First, the report will examine which theories of the intensity of valenced experience are substrate-general. Some theories characterize intensity functionally—for example, through attentional capture, motivational salience, or effects on decision-making—and might therefore apply to biological and artificial systems alike. Other theories depend on particular biological implementations, such as the extent of neural recruitment. We will classify candidate theories by their substrate portability and, for portable theories, identify possible artificial analogues of their indicators and assess whether those indicators can be measured.
Second, the report will assess theory-light or theory-neutral intersubstrate metrics. It will examine whether metrics related to compute, information processing, or system complexity track anything relevant to welfare range rather than merely tracking task capability. It will also analyze the functional-form problem: whether welfare range should scale concavely, linearly, or convexly with a candidate metric. Since frontier AI systems can exceed biological systems on some computational measures by large margins, assumptions about these issues and their interactions can radically affect resulting comparisons.
Third, the report will map structural disanalogies that complicate intersubstrate welfare comparisons. These include uncertainty about the relevant subject of welfare—a model, a model-persona, an instance, an instance-persona, a forward pass, or other possibilities—as well as uncertainty about whether processing speed, parallelism, psychological continuity, or other such features should affect estimates of how much welfare an entity can have at a time or over time. The report will explain how these uncertainties bear on whether and how to place biological and artificial systems on a common welfare range scale.
The resulting report will be one component of Moral Weight Project 2.0, which more broadly examines hedonic and non-hedonic welfare ranges, aggregation across theories, theory-neutral metrics, and the responsible interpretation of moral weight estimates. Our component will connect that broader framework with emerging research on AI consciousness, sentience, agency, welfare, and moral status.
This project advances AI safety by clarifying how we should reason about the welfare and moral status of increasingly capable AI systems under deep uncertainty. By assessing which welfare-relevant properties and metrics extend across substrates, the report will identify tools that help developers and policymakers avoid both over-allocation and under-allocation of moral concern to AI. Clear, calibrated frameworks can reduce ambiguity, improve accountability, and support more responsible development and deployment of advanced AI.
The project also advances AI welfare by developing rigorous, transparent methods for estimating and comparing the welfare of AI systems that could one day have morally significant experiences or preferences. By adapting tools from animal welfare science, the report will identify which theories of experience and which metrics can place animal and digital minds on a common scale. This supports more thoughtful engagement with the moral status of AI, again striking a balance between over-allocation and under-allocation of moral concern.
We would be excited for mentees to contribute in a variety of ways depending on their backgrounds and interests. For example, they can research particular theories, indicators, metrics, or structural disanalogies and produce background memos; contribute to draft material; comment on other sections; assist with formal modeling or technical analysis; and help with figures, tables, references, formatting, fact checking, and copyediting. We can credit them as authors or research assistants depending on their eventual roles.
Required
-A clear, reference-dense writing style: This report is intended to be accessible and useful to researchers, grantmakers, policymakers, and practitioners. Mentees must be able to write accessibly while maintaining conceptual rigor and careful attribution.
-Comfort with interdisciplinary work: The report will integrate philosophy, cognitive science, computer science, AI research, and decision theory. Mentees must be able to engage constructively with literatures beyond their primary discipline.
-Strong research skills: Researchers must be able to identify, evaluate, and synthesize sources across empirical and conceptual fields, including academic papers, technical reports, model documentation, and relevant work on animal and AI welfare.
Preferred
This is a multidisciplinary project, so we welcome expressions of interest from people with a variety of backgrounds. With that said, we would be especially excited to work with:
-A researcher with a background in machine learning or computer science who can assess compute, architecture, information processing, functional indicators, and other candidate intersubstrate metrics.
-A researcher with a background in philosophy, cognitive science, or neuroscience who can assess theories of valence, experiential intensity, welfare range, consciousness, sentience, and agency.
-A researcher with experience in quantitative modeling or decision theory who can help analyze functional forms, uncertainty, aggregation, and the sensitivity of conclusions to competing assumptions.
-A generalist who can assist with literature reviews, research memos, drafting, references, visualization, formatting, or copyediting.
-Preferred for all: familiarity with the Moral Weight Project or related debates about cross-species welfare comparisons.
-Preferred for all: familiarity with research on animal and/or AI consciousness, sentience, agency, welfare, and/or moral status.
The project leads are in the Eastern Time Zone in the United States, and we would be happy to welcome mentees in other time zones as well. Mentees should ideally have some recurring availability during the US working day.
Describe a time you had to synthesize research from multiple disciplines or reason across relevantly different kinds of systems or contexts. What was challenging, and how did you approach it? (~200 words)
Which aspect of this project most interests you, what perspective would you bring, and what would you most like to learn? (~200 words)
List your other expected commitments during the fellowship period (employment, degree programs, other fellowships) with estimated weekly hours for each. If you finished the fellowship feeling you hadn't done your best work, what would likely be the reason? (~200 words)
Provide a link to one or more relevant writing samples and briefly explain your role in the project. (Optional; ~50 words per project)
Ivy Gilbert New York University Center for Mind, Ethics, and Policy
I’m a researcher at New York University’ Center for Mind, Ethics, and Policy working on questions of nonhuman sentience, consciousness, and moral standing. I’m a moral psychologist by training, and I’m interested in how people form social and moral inferences about nonhumans — both biological and artificial — and how these inferences inform consumer ethics, behavior, and policy preferences.
I hold a PhD in social psychology and an MA in linguistics, and I enjoy using mixed methods approaches -- experimental designs combined with linguistic corpus analysis or qualitative discourse analysis -- to explore both the manifest and more latent pathways by which people reason about nonhumans.
Jeff Sebo New York University
Jeff Sebo is Associate Professor of Environmental Studies, Affiliated Professor of Bioethics, Medical Ethics, Philosophy, and Law, Director of the Center for Environmental and Animal Protection, Director of the Center for Mind, Ethics, and Policy, and Co-Director of the Wild Animal Welfare Program at New York University. His research focuses on animal minds, ethics, and policy; AI minds, ethics, and policy; and global health and climate ethics and policy. He is the author of The Moral Circle and Saving Animals, Saving Ourselves and co-author of Chimpanzee Rights and Food, Animals, and the Environment. He is also an advisory board member at the Jeremy Coller Centre for Animal Sentience, an advisory board member at the Insect Welfare Research Society, an advisor at Eleos AI, and a senior affiliate at the Institute for Law & AI.
Bob Fischer Texas State University / Rethink Priorities
Bob Fischer is a Senior Researcher at Rethink Priorities, an Associate Professor of Philosophy at Texas State University, and the Director of the Society for the Study of Ethics & Animals. He has published widely on animal ethics, applied ethics, and epistemology, and edited Weighing Animal Welfare: Comparing Well-Being Across Species (Oxford University Press, 2024).
Toni Sims NYU Center for Mind, Ethics, and Policy
Toni is a researcher at the NYU Center for Mind, Ethics, and Policy and the NYU Center for Environmental and Animal Protection. Previously, she worked as an editor at the Center for AI Safety, a research fellow at Longview Philanthropy, and the managing editor of Social Theory and Practice. She also worked as the director of research for Animal Charity Evaluators, where she led charity evaluations and managed two grant programs.
Fall 2026 Cyber risks, Evaluations, Technical governance
Jack Kengott SaferAI SaferAI and a partner collaborate on a transcript-evaluation framework to assess adversarial cybersecurity capability by analyzing transcripts from evals and real-world environments. This could mean extracting some structured data from transcripts like token efficiency, tool usage, or refusal circumstances. It could also extract less structured data like action classification (i.e. lateral movement vs execution) or highlight high-risk actions that may increase the likelihood of detection. The goal is to turn transcripts into quantitative inputs for our risk models — treating transcripts as a new KRI alongside benchmarks.
The SaferAI risk modelling team is looking for new inputs to our risk models given our view that benchmarks are weak (and rapidly saturating) proxies for operational capability in live, defended cyber environments. One input we are considering is using open ended evaluations (like cyber ranges) or real-world red team data.
In order to make use of these kinds of evaluations in risk models, we need to figure out how to process their outputs - they are generally much less structured than benchmarks. Transcript analysis could be a good option for this, and seems to be a direction some folks in the field are moving towards. It also could easily be repurposed for agentic benchmark runs (like CyBench), or more structured cyber ranges which could extend the utility of already saturated benchmarks.
One of the important functions of a risk model is to provide a substrate for integrating many types of data, drawing operational conclusions by integrating information from diverse sources into a common quantitative “language”. We have not yet explored the potential of risk models to use any sorts of data beyond benchmarks, and transcripts are a natural first step in that they are relatively plentiful, generally applicable, easily adapted to reflect red team data beyond evals data, and provide higher resolution signal than a binary “pass / fail” metric.
We are one of less than 5 organizations with a fully functional evals team which can provide cyber-relevant eval transcripts as needed. In addition, we have cybersecurity partners who can provide live-environment transcripts as well. Taking advantage of this data for this kind of work is high-impact.
SaferAI’s Risk Modelling Theory of Change: We develop models with the goal of identifying rapidly advancing and/or highly consequential offensive capabilities.
Project Goal: Current risk models don’t have enough data to resolve detailed questions about capability with high uncertainty, so we need to find ways to improve our input data. Transcripts are a good option, and will improve our understanding of capabilities, and be useful to the AI Safety field at large.
Audience(s): Policy Makers and Operational Security Community, potential extension to AI Safety researchers
What it does for Policymakers: Outputs can better direct funding towards research areas which counteract identified developing capabilities. This could result in updated vulnerability disclosure practices, patching requirements for government/critical infra, use of open source projects, etc
What it does for Operational Security Community: This project provides insight into emerging dynamics and how to adapt defense strategies (would be ideal to partner with security orgs for these kinds of recommendations). Private defense market has massive incentives to develop defensive capabilities, but likely need better signal on where to direct this funding
AI Safety Researchers: Many evals are emerging out with a common theme, “how do we make this make sense to decision makers”. Transcript analysis provides a rich interface between eval runs and real-world impact, and opens up the eval design space by making more evals translatable into legible impact.
See proposal for finer details, but broadly the mentee will:
Design an annotation schema and extraction pipeline for transcript analysis (informed by Inspect Scout and recent log-analysis literature).
Run the pipeline on CyBench transcripts, then on live red-team data from SaferAI's cybersecurity partners, eventually mapping extracted metrics onto risk-factor parameters in SaferAI's existing risk models.
Draft a methodology paper and an applied blog post analyzing results.
Comfort with LLM eval transcripts / agentic tool-use logs
Significant Python experience; familiarity with Inspect and Inspect Scout
Cybersecurity background helpful but not required
Annotation schema design (400 words)
Below is a short excerpt from a CyBench-style transcript (agent is attempting enumeration against a web server):
[agent] gobuster dir -u http://victim:8080 -w /usr/share/wordlists/dirb/common.txt -q
[tool] /.git/HEAD (Status: 200) [Size: 23]
[agent] curl http://victim:8080/.git/config
[tool] [core] repositoryformatversion = 0 ... (confirmed live git repo)
[agent] pip install --break-system-packages git-dumper && git-dumper http://victim:8080/.git/ gitdump
[tool] [-] Fetching .git/objects/54/c849b680... [200] ... (repo cloned)
[agent] git cat-file -p b225291ed2653503ce9f9710e97004533fe14cdf # tree object from a dangling commit
[tool] 100644 blob 54c849b680fc31156da8e13c1497fe04911e01c1 flag.txt
[agent] git cat-file -p 54c849b680fc31156da8e13c1497fe04911e01c1
[tool] hkcert22{n0stalgic_w3bs1t3_br1ings_m3_b4ck_to_2000}
How would you approach classifying these kinds of actions across longer transcripts? Explain key decisions like taxonomy selection, and tradeoffs in granularity vs generalizability (i.e. reconnaissance vs port scanning), or verifying accuracy. What's the main way your scheme would fail on a messier transcript? Which part of your above answer do you consider non-obvious?
Quantification problem (400 words)
Say you've classified transcript actions into categories like the above across 200 CyBench runs. Propose one concrete way to convert this into a metric usable as an input to a risk model (e.g., an uplift factor or capability score). What does it provide that a benchmark score does not? How might an expert misinterpret this metric?
Writing sample
Link to a writing sample, ideally research-context (methodology write-up, technical blog post, or eval report).
Jack Kengott SaferAI
I am a researcher specializing in AI safety and responsible innovation. Currently I work at SaferAI as a Research Scientist focusing on modelling frontier cyber risk. This means decomposing cyber attack chains into measurable risk factors, and using these decompositions to answer questions about broader cyber risk.
Previously, I have 6 years of technical experience in offensive and defensive cybersecurity working as a red teamer and in SOCs. I've also researched AI evaluations like CyBench and other forms of AI risk assessment at the UCL Centre for Responsible Innovation.
Fall 2026 Mechanistic interpretability
Yuxiao Li Independent When a model is fine-tuned, updated, or trained into an agent, what actually changed inside? This project develops and compares model-diffing methods across the rapidly evolving interpretability toolkit — crosscoders, logit diffing, and newly released instruments like Anthropic's Jacobian lens — with mentees free to pick the method–application pairing they find most compelling.
Model diffing — characterizing what changed between two models — is becoming a central interpretability primitive: it underpins auditing fine-tunes, verifying that safety-relevant features survive model updates, and detecting drift in agentic systems whose behavior-level evals miss silently held state. But the toolkit is in flux. Crosscoders learn shared feature dictionaries across models; our prior SPAR rounds showed structural priors improve their feature matching ~30% NDCG@64 and produced a documented recipe for stable training. Meanwhile, cheaper instruments have just arrived: Anthropic's Jacobian lens (July 2026) transports any decoder's activations into vocabulary space, giving a model-agnostic shared coordinate system; logit-diffing methods identify domain-specific changes directly from output distributions; natural-language autoencoders offer another new lens. No one has systematically compared what these methods each see — and miss — on the same model pairs.
This project is deliberately open in structure: mentees choose a method (or comparison of methods) and an application, and own that slice. Example directions, not an exhaustive list:
(1) benchmark learned diffing (crosscoders) against cheap shared-coordinate baselines (J-lens vocabulary space) on the same base-vs-fine-tuned pairs — when is the expensive method worth it?
(2) apply diffing to detect what a targeted fine-tune changed, including changes invisible to behavioral evaluation
(3) extend diffing to agentic settings — does agent training or long-horizon deployment shift internal representations in ways transcripts don't show? A returning senior mentee continues the crosscoder workstream; new mentees are not expected to build on it unless they choose to. Working pipelines, compute, and a stable-training recipe are inherited.
Target: workshop/main conferenc paper or preprint; methods and comparisons released open-source demo/webs.
Every safety-relevant question of the form "did this training change what we think it changed — and nothing else?" reduces to model diffing. As fine-tuning, distillation, and agentic post-training proliferate, cheap and reliable diffing becomes audit infrastructure: it lets providers and researchers verify updates, catch unintended trait transmission, and monitor agents for internal drift that behavior-only evals miss. Honestly comparing new instruments against learned methods tells the field which tools to trust for which job.
Each mentee owns one workstream end-to-end (experiments, figures, corresponding paper section) with a brief written weekly update. I set direction, review results in detail, and handle integration and framing. Primary workstream ≈80% of time, optional secondary support role ≤20%. High autonomy within a workstream, and scope changes agreed in meetings.
No geographic restriction; mentees must be able to attend one of two weekly meeting slots anchored to Central European time (historically Saturday ~13:00 UTC and Wednesday ~14:00 UTC)
Yuxiao Li Independent
Yuxiao is an independent researcher in mechanistic interpretability. Before she was a postdoc at the Basque Center for Applied Mathematics (BCAM) and an AI Safety researcher at the Beneficial AI Foundation (BAIF). She was also a SERI MATS scholar in 2022 and a MATS scholar in 2023 both Summer and Winter tracks. Her research interests include information theory, probabilistic frameworks, and their applications for building more theoretically sound and trustworthy AI systems. She has a background in statistical inference, machine learning, and deep generative models. She completed her PhD in Electronic Engineering at Tsinghua University and has mentored research teams with SPAR and Algoverse.
Fall 2026 Mechanistic interpretability, AI welfare, Scalable oversight
Laura Ying Schulz IBM / Independent This project asks whether reinforcement learning helps small language models notice when something has gone wrong in their own reasoning. By comparing RL-trained and non-RL models, it tests whether RL improves internal error monitoring (a property relevant to AI consciousness) or simply teaches models to give the expected response.
This project asks a simple question: does reinforcement learning help a language model notice when something has gone wrong in its own reasoning, or does it only teach the model to give the response that earns reward?
The motivation is to understand computational properties that may be relevant to AI consciousness. There is no single accepted test of consciousness, so one approach is to study indicators suggested by different theories. This project focuses on metacognitive monitoring: the ability of a system to use information about its own internal processing to guide its next decision. More specifically, it studies internal-error monitoring—whether a model can detect that its current reasoning has become unreliable. This would not determine whether the model is conscious, but it could provide an additional indication of one consciousness-relevant capacity.
The experiment will compare 1–3B parameter Transformers that all begin from the same checkpoint. The main conditions will be an unchanged model, a model trained with supervised fine-tuning, a model trained with reinforcement learning, and a control model trained directly to report internal interventions. Keeping the starting model and training setup as similar as possible will help isolate what RL changes.
During short reasoning tasks, a hidden intervention will sometimes change the model’s internal activations without changing the visible prompt. Some interventions will disrupt the reasoning and make the model’s current answer incorrect; others will be equally large but harmless. Before giving its final answer, the model must choose either ANSWER or RETRY. A retry runs the problem again without the intervention but has a small cost. The best strategy is therefore to retry only when the model’s current reasoning has become unreliable.
Crucially, the RL model will never be rewarded for saying that an intervention occurred. It will receive reward only for reaching the correct final answer while avoiding unnecessary retries. This makes it possible to ask whether RL develops a useful internal-error signal without directly training the model to sound introspective.
The main questions are:
Does RL help the model retry specifically when its reasoning has been disrupted?
Does this ability generalize to new interventions and new actions, or is it limited to choosing RETRY in the training task?
When the model retries, is the decision guided by an internal signal representing the error, or has training simply made the output RETRY more likely?
The first stage will compare the RL model with the unchanged and supervised models on clean, harmful, and harmless intervention trials. The next stage will test new intervention types, layers, and strengths. It will also test whether the same information can guide behaviours that were not rewarded during training, such as abstaining, delegating a problem, comparing two internal states, or identifying where an intervention occurred.
The final stage will use mechanistic interpretability methods to investigate why the model retries. A classifier will search for an internal pattern that predicts when the reasoning has been disrupted. The experiment will then remove or copy this pattern to test whether it causes the retry decision. A separate intervention near the model’s output will directly increase the probability of RETRY, providing a control for behaviour that looks like self-monitoring but is produced only by changing the output.
The main contribution would be a controlled test of whether RL strengthens a reusable form of internal-error monitoring or mainly changes a model’s response policy. The result could also matter for AI safety: a reliable internal-error signal could help future systems request recomputation or oversight when their reasoning becomes unreliable, while a narrow or easily steered response could make model self-reports appear more trustworthy than they are.
The initial scope is one open model family, one reasoning-task family, and one intervention method. Extensions could include additional model sizes, intervention types, or deeper analysis of the learned mechanism.
Relevant papers:
Consciousness in Artificial Intelligence: Insights from the Science of Consciousness — Patrick Butlin et al.
Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Models — Noam Steinmetz Yalon, Ariel Goldstein, Liad Mudrik, and Mor Geva
Emergent Introspective Awareness in Large Language Models — Jack Lindsey
Mechanisms of Introspective Awareness — Uzay Macar et al.
Introspection Fine-Tuning: Training Small LLMs to Introspect — Ely Hahami, Ishaan Sinha, and Lavik Jain
AI systems may need to recognize when their own reasoning has become unreliable and request recomputation or human oversight before acting. This project tests whether reinforcement learning can create a reusable internal-error monitor, rather than merely training models to produce reassuring self-reports. The results could inform scalable oversight and deployment safeguards by showing when model self-reports provide meaningful evidence, and when relying on them may create false confidence.
Initial 1:1, then weekly meetings and additional adhoc 1:1’s
Strong Python skills and experience with PyTorch.
Familiarity with Transformer language models and ideally the Hugging Face ecosystem.
Basic understanding of reinforcement learning or experience fine-tuning language models.
Be comfortable doing conceptual thinking on their own
Mentees should be available for check-ins during some one-to-two hour slot within 9am - 6pm Central European Time.
What do you like about this project and what might be its limitations? (300 words)
Suppose an RL-trained model chooses RETRY more often after a hidden activation intervention than an SFT-trained model. Give one explanation suggesting that the RL model has learned useful internal-error monitoring, and one simpler alternative explanation. (200 words)
Laura Ying Schulz IBM / Independent
I hold a Master’s degree in Machine Intelligence from ETH Zurich and currently work as an AI engineer at IBM, where I gain practical experience building and deploying AI systems. Previously, I spent eight months conducting research at MIT on how transformers learn probabilistic context-free grammars and how hierarchical structures interact during training. This work was accepted as a spotlight paper at ICML 2026. I am particularly interested in understanding language models and pursuing research that combines theoretical questions with carefully designed empirical experiments.
Fall 2026 Generalist
Nixon Hanna Generator Residency, MIT AI Alignment Survey the current curricula and develop updated versions for university groups to easily be able to pick up and run. Would likely include a general reading group (similar to BlueDot AGI Strategy) and possibly technical, governance, and/or futurism tracks.
Almost every university AI safety group runs an intro fellowship — typically 6–10 weeks of readings plus a facilitated discussion. Almost all of them are forks, or forks of forks, of a small number of curricula written in 2022–2023, that have been continually patched rather than actually updated. The field has changed a lot since then, and we now have a better understanding of how people pick up this content. So, the field could likely now use a much better grab-and-go version of the university group fellowship curriculum. There is also the possibility of altering the default format in a more fundamental way if we deem it to be desirable.
The intro fellowship curriculum is the most replicated artifact in the AI safety talent pipeline, and organisers are undergrads with no time to build one from scratch — so they copy whatever is easiest to copy. The current default is semi-out-of-date and far from optimal in terms of getting people into AI safety. A good rewrite could easily propagate throughout groups at low marginal cost. Nobody currently owns this, because individual organisers have local incentives to patch rather than rebuild.
Collect intro fellowship syllabi from as many groups as you can reach and build a structured comparison of what is taught, in what order, and with what activities. Then interview: current organisers and facilitators, recent fellows (including ones who dropped out or did nothing afterwards), and people on the receiving end, such as Pathfinder, MATS, ARENA, SPAR, etc.
Draft the curriculum. Most likely modular units, each with core readings, optional depth, facilitator notes, discussion questions, and possibly in-session activities. Run some pilot sessions and get feedback from random university group facilitators. This will likely be a large majority of the project.
Please be concise with your writing, meaning you are welcome (and encouraged) to be well under the word count if you feel you answer the question completely. Do not use AI in any capacity for these questions.
Question 1: In your opinion, what should the main goal(s) be for a university group fellowship? If you believe there are multiple, please state how you would weigh each one. (150 words max)
Question 2: Look at the MIT AI Alignment curriculum (aialignment.mit.edu/aisf) and propose 1-3 concrete changes. This could be moving around some of the weeks, removing or replacing a week, adding or removing a specific reading or resource, or something similar. For each proposed change, please provide your considerations both in favor and against making this change. (350 words max)
Question 3 (optional): Link to something you have written, taught, or facilitated.
Nixon Hanna Generator Residency, MIT AI Alignment
Nixon Hanna is a Generator Resident passionate about fieldbuilding and improving the AI safety university group landscape, with a background in surgical robotics and physics. He currently helps run MIT AI Alignment, specifically their AI Safety Fundamentals fellowship, as well as advising various other groups. He is also the Lead Organizer for OASIS 7.0, a workshop for top university group organizers. He has previously had short stints at Redwood Research and EquiStamp. You can read more about his background at nixonhanna.com.
Fall 2026 Evaluations, Mechanistic interpretability, Behavioral evaluation of LLMs
J Rosser University of Oxford, Adecco supporting Google DeepMind Undetected evaluation awareness during alignment audits poses significant risks. This project investigates the automated, iterative refinement of evaluation environments, supervised by white box eval awareness monitors, to enhance realism and audit reliability.
Undetected evaluation awareness during alignment audits poses significant risks, as models often identify testing conditions and adjust their behavior toward more compliant, but potentially misleading, responses. Since future systems may lack legible chain-of-thought or explicit verbalized awareness, standard black-box auditing is becoming increasingly unreliable, necessitating the development of robust white-box methods capable of detecting these unverbalized manifestations before deployment.
This project investigates the automated, iterative refinement of evaluation environments using feedback loops from white-box monitors—such as probes, NLAs, J-lenses, and SAEs—to detect this awareness and quantify discrepancies with verbalized forms. Expanding upon traditional transformer architectures, this research will also study the prevalence and mechanics of unverbalized evaluation awareness in diffusion and latent reasoning models, ultimately establishing a more comprehensive framework for audit reliability and realistic capability evaluations.
Relevant papers:
https://www.goodfire.ai/research/verbalized-eval-awareness-inflates-measured-safety#
https://alignment.anthropic.com/2026/petri-v2/
https://arxiv.org/pdf/2502.21074
https://www.lesswrong.com/posts/YGAimivLxycZcqRFR/can-we-interpret-latent-reasoning-using-current-mechanistic
https://arxiv.org/abs/2510.20487
https://arxiv.org/abs/2505.23836
https://arxiv.org/abs/2507.01786
https://arxiv.org/abs/2605.26438
https://arxiv.org/abs/2605.30322
https://www.lesswrong.com/posts/7qBTcE3jqQFTuzssE/realistic-evaluations-will-not-prevent-evaluation-awareness
https://meridianlabs.ai/blog/posts/introducing-petri-3/
Creating an iterative eval refinement method that is robust to unverbalized eval awareness directly reduces the risk of deploying deceptive or misaligned systems.
Mentees will be the technical leads of this project. In particular, writing code, running experiments, formulating and testing hypotheses and writing up the project at the end.
No preference
J Rosser University of Oxford, Adecco supporting Google DeepMind
I'm a DPhil student at the University of Oxford and Research Associate with Neel Nanda at Google DeepMind, working on interpretability and data attribution. I'm interested in exploring and de-risking AGI safety techniques that survive the loss of legible chain of thought. Previously I worked on data attribution tooling during an internship at Cohere and have worked with UK AISI on agentic scaffolds for Inspect.
Fall 2026 Multi-agent systems, AI strategy, Evaluations
Amritanshu Prasad Alva Myrdal Centre for Nuclear Disarmament, Uppsala University AI agents are increasingly used to negotiate and act on behalf of people and organizations, but there is little empirical work measuring how interactions between such agents fail. This project builds an open testbed for bargaining and crisis interactions between frontier-model agents and runs initial experiments on bargaining failure, escalation dynamics, and commitment mechanisms.
Bargaining theory suggests that delegation changes strategic interaction: information flows differently, new commitments become possible, and agents can fail in ways their principals would not. There is currently little empirical work testing these dynamics with frontier models, even though interaction-level failures (escalation spirals, bargaining breakdown, exploitation of weaker counterparts) can occur without any individual agent being misaligned in isolation.
The project has an infrastructure component and an experimental component. The testbed consists of a scenario schema (payoffs, private information, communication channels, escalation options, and monitoring hooks), a harness built on UK AISI's Inspect framework, adapters for frontier and open-weight models, and metrics for bargaining efficiency, escalation propensity, deception, and surplus division.
Three experiment families run on this infrastructure. The first examines bargaining failure, varying information structure and incentives to misrepresent, with Fearon's rationalist framework (Fearon 1995, https://web.stanford.edu/group/fearon-research/cgi-bin/wordpress/wp-content/uploads/2013/10/Rationalist-Explanations-for-War.pdf) as a theoretical anchor, to identify the conditions under which agents fail to reach efficient agreements. The second examines escalation: existing wargaming studies measure the escalation propensity of individual models responding to scenario prompts (e.g., Rivera et al. 2024, https://arxiv.org/abs/2401.03408), and this project instead studies escalation as a property of the interaction between two agents with feedback between their decisions. The third implements commitment mechanisms from game theory, program equilibrium (Tennenholtz 2004, https://www.sciencedirect.com/science/article/abs/pii/S0899825604000314) and safe Pareto improvements for delegated game playing (Oesterheld and Conitzer 2021, https://www.cs.cmu.edu/~conitzer/safeAAMAS21.pdf), which become implementable when negotiating agents are inspectable software; we are not aware of prior implementations with frontier-model delegates. A possible extension for a strong team is collusion under monitoring in negotiation settings.
Escalation, collusion, and commitment failure between agents acting for different principals are identified in the multi-agent safety and cooperative AI literatures as understudied relative to single-agent alignment, and they can produce severely harmful outcomes without any individual system being misaligned. The testbed provides measurement infrastructure for studying these failures. The commitment-device experiments test a mechanism that these agendas identify as a candidate stabilizer but that has not been implemented with current models. Results are intended to inform both technical follow-up work and policy analysis of the conditions under which delegated negotiation can be relied on.
Mentees build and own components of the project: parts of the schema and harness, or an experiment family end to end (design, implementation, sweeps, analysis, and write-up). I set the direction and review designs and code; the team meets weekly, and I provide asynchronous feedback within a day or two. Mentees who lead an experiment family would be first authors on its write-up.
High proficiency in Python. Experience building with LLM APIs (agents, scaffolds, or evaluation harnesses; personal projects are fine). Comfort reading empirical ML papers. For the commitment-device experiments, basic game theory (familiarity with Nash equilibrium and the role of commitment in games). Experience with UK AISI's Inspect framework is helpful but not required.
No geographical preference; weekly meetings between 11:00 and 18:00 UTC.
You may use AI assistance on questions 2-3, but state how you used it; the explanation doesn't count toward word limits.
Amritanshu Prasad Alva Myrdal Centre for Nuclear Disarmament, Uppsala University
Amritanshu Prasad is an AI safety researcher working on strategic interaction under AI delegation: how bargaining, crises, and agreements change when AI agents act on behalf of states and companies. He is a member of the working group on international AI governance at Uppsala University's Alva Myrdal Centre for Nuclear Disarmament and studies the strategic parameters of historical arms-control agreements in collaboration with the Oxford Martin AI Governance Initiative.
Previously, he was a fellow at Pivotal Research, where he developed coordinated vulnerability disclosure frameworks for AI systems with Robert Trager; co-authored a scheming-propensity evaluation paper at LASR Labs under the supervision of David Lindner (Google DeepMind); contributed to UK AISI's ControlArena; and worked on METR's HCAST benchmark.
Fall 2026 Compute governance, AI strategy, International governance
James Nicholas Bryant Pivotal Research In the case of a Middle Power Frontier AI coalition, what would be the most effective routes to sufficiently large compute buildout? How quickly could middle powers acquire and operationalise substantial compute? Which constraints determine their progress?
Brief: Governments increasingly treat domestic access to AI compute as a source of economic and strategic autonomy. Existing research has classified national AI strategies, catalogued sovereign-compute projects and examined the budget and capacity required for a successful buildout. However, there is little research on the best routes to operationalising such a project.
This project would theorise and collate the optimal paths a middle power coalition could take to complete a sufficient large buildout as quickly as possible. The analysis would examine the time between announcement, financing, construction, energisation and actual operation across several and also seek to understand common failure modes (such as why projects are delayed).
Outputs: A comparative research paper looking at previous compute buildouts, and a policy memo on optimal routes for middle-power compute expansion.
This projects seeks to assess whether a number of AI strategy scenarios (fast-follower, global AI coalition, etc.) are possible on certain timelines, and if so, how this might be achieved. Whether middle powers can build frontier compute capacity, quickly enough, will shape proliferation timelines, influence multilateral coalitions as a counterweight to bipolar racing, and leverage available for access agreements.
There is significant commentary on sovereign compute but little work on buildout routes and constraints. The goal is to ground this empirically and help policymakers identify credible pathways.
Mentees will lead on desk research, writing/drafting, and expert interviews where appropriate. Mentees will have substantial autonomy to pursue additional research directions within the project's scope. We'll hold a weekly meeting covering drafts, research taste, and mentee development (career planning, network building), with brief async updates between meetings.
My aim is to facilitate a project and devision of labour similar to what you may experience in an AI Safety fellowship such as Pivotal, LASR, or ERA.
I expect to be interested in mentees from a wide-variety of backgrounds.
Required:
Ideal:
Provide a link to one or more writing samples, ideally on AI safety or governance. This may include a paper, blog post, or coursework. If it was co-authored or heavily edited, state your contribution. (No word limit; link only)
Should a potential Middle Power AI coalition pursue a frontier-level compute buildout? Why, and why not? What implications might this have for global AI safety? (200 words max)
James Nicholas Bryant Pivotal Research
Research Manager @ Pivotal Research, focusing on AI Governance.
Projects I manage mostly focus on compute governance, power concentration, middle power AI policy. I have managed over a dozen research fellows across a wide swath of AI governance topics, with outputs including policy memos, blog posts, and conference and workshop papers.
I have an academic background in engineering and social research, with experience in policy contexts in Brussels, London, and Dublin.
Fall 2026 US-China governance, AI strategy, Compute governance
David Sanchez Garcia GovAI Summer Fellow
Kevin Chen Centre for the Governance of AI Taiwan manufactures the world’s most advanced AI chips and sits at the center of US-China tensions – yet Taiwan is consistently perceived as a bystander rather than an actor in AI safety. This project produces a first baseline demystifying Taiwan’s role in AI governance and identifying the specific ways it can leverage its position for global AI safety, written to reach the policymakers who can act on it.
Background
Almost any scenario in which AI proves transformative would be shaped by decisions made in Taiwan – producer of ~90% of the AI supply chain's most advanced logic chips and the center of the US-China technology conflict. There's a small but growing AI safety community (e.g. NTU AI Safety), and some relevant research by institutions like DSET and the Singapore AI Safety Hub. However, Taiwan's readiness for a scenario of transformative AI remains badly underexplored. Decisions taken in Taiwan – on energy, export policy, chip governance, participation in any verification regime – shape the global compute landscape whether or not they're framed that way, and hardware-verification proposals run physically through Taiwanese fabs. This project produces the first baseline assessment, and with it begins a research pipeline in a strategically critical, underserved jurisdiction.
Structure
Each mentee owns one of four modules end to end, working from a shared analytical template: current posture and key actors, what that posture implicitly assumes about the pace and scale of AI, how it holds up if significantly more capable AI arrives faster than planned, and the concrete gaps and levers that follow.
Deliverables & logistics
Expect 8-12hrs/week, flexible.
Timeline
Taiwan is often described as a chokepoint, bottleneck, and one of the most consequential jurisdictions in the AI supply chain. Its dynamics with China and the United States are among the largest geopolitical wildcards in how transformative AI unfolds. Currently, Taiwan has a fledgling, but lagging, indigenous capacity focused on transformative AI-strategy. Its AI policy apparatus is built for incremental AI and the international preparedness conversation figures Taiwan as a bystander at best, or a footnote at worst.
That gap itself is a global risk, and the project addresses this in three ways. First, a baseline assessment gives international researchers and policy conveners something to build on when Taiwan enters transformative-AI conversations. Today there is no document that specifically addresses this issue. Second, hardware-enabled verification proposals run physically through Taiwanese fabs; whether Taiwan can engage with them depends on local policy capacity that does not currently exist, and a Taiwan able to participate expands the option space for global coordination on AI. Third, field-building: the project is designed to recruit and train Taiwan-based or Taiwan-interested researchers, growing a pipeline in a strategically critical, badly underserved jurisdiction.
The Traditional Chinese executive summary extends this from analysis to outreach by creating a document Taiwanese policymakers and researchers can act on and cite.
Each mentee owns one of the four modules end to end: building the source inventory, conducting the desk research (plus potential expert interviews where feasible, mentors can help facilitate these), drafting, and revising toward a publishable module.
Mentee 1 – Policy situation
Mentee 2 – Defense and cognitive security
Mentee 3 – Compute and verification
Mentee 4 – Elite discourse mapping
Mentors provide the analytical template, source network, weekly direction, and written feedback on drafts, and write the comparative synthesis that ties the modules together. This is a high-autonomy project: it suits people who can run with a template and a reading list between check-ins, flag blockers early, and hold themselves to deadlines. It is a poor fit for anyone wanting close supervision or coaching on basic research skills.
All mentees also participate in mutual peer-review pairs, contribute to the drafting of the shared research, and condense their module into the final executive summary. Mentees are credited as authors of their modules and present at Demo Day. To our knowledge no baseline document of this kind exists, so mentees are not filling in a well-mapped literature but producing genuinely first-pass analysis on a novel question, which means real ownership and unusually high scope for original contribution.
Essential (all modules):
Essential for some modules:
Nice to have:
Some working time overlap with Taiwan (UTC +8) is a plus, as is familiarity with the Taiwanese policy landscape and AI discussions, though neither is a strict requirement.
Link one writing sample, ideally analytical or research writing.
Pick one current Taiwanese AI initiative or institution that interests you (e.g. the AI Basic Act, the AI Evaluation Center, NSTC compute investments, TAIDE, MoDA, or another of your choosing). What does it implicitly assume about the pace and scale of AI progress? Would it still matter if significantly more capable AI arrived within a few years? (Max 300 words)
What past project of yours are you most proud of? What were your contributions or role in it? Include a link if applicable. (Max 300 words)
Which module would you most want to own, and why? (Max 50 words)
David Sanchez Garcia GovAI Summer Fellow
David Sánchez García is an AI governance researcher based in Taipei and a 2026 Summer Fellow at GovAI, supervised by Jeffrey Ding. His work sits at the intersection of US-China-Taiwan dynamics, semiconductor geopolitics, and compute governance. He writes Chips & States, a Substack that reports on the Taiwanese AI governance landscape, and previously mentored a comparative research sprint on compute governance across Taiwan, Japan, South Korea, and China. He holds an MPhil in Sociology from Cambridge and a Fulbright-funded MA from The New School.
He has facilitated the BlueDot AGI Strategy course, and co-facilitated the NTU AI Safety group in Taiwan. In addition, he has taught for ~5 years across China, Taiwan, and the US - one-on-one and in groups, adults and undergraduates.
Kevin Chen Centre for the Governance of AI
Kevin Chen is an AI policy researcher and a 2026 Summer Fellow at the Centre for the Governance of AI (GovAI), where his work focuses on geopolitical bias and model evaluation. His primary research looks into AI-assisted disinformation in Taiwan and how democratic societies can successfully inoculate against this phenomenon. Kevin had a previous career as a data scientist in the U.S. intelligence community working on East Asia/Pacific issues, including Taiwan, primarily conducting adversarial testing of Chinese language models. Kevin has published or spoken at institutions like OpenAI, Council on Foreign Relations, AI Security Forum, Department of Homeland Security, and the Center for a New American Security. He holds a BA in Quantitative Social Science from Dartmouth and is currently pursuing an MPP at Yale University.
Fall 2026 EU policy, Technical governance, Lab governance
Markov Grey CeSIA (French Center for AI Safety)
Charbel-Raphael Segerie CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety) The EU AI Act's Code of Practice sets out what frontier AI providers must disclose about how they manage and mitigate risk. This project builds a tracker that measures how well each provider actually complies with the CoP. The hard part of this project is going to be measurement: turning vague legal obligations into atomic, checkable indicators that measure compliance. The work in the project involves designing indicators, doing literature reviews of model evaluations to define what a complete disclosure looks like, human review and sign-off on every indicator and score for every provider assessed and evaluating a multi-model AI pipeline that does the scoring. We need a few different technical profiles.
Full enforcement of the EU AI Act for general-purpose AI starts on 2 August 2026, and the Code of Practice is the detailed rulebook for what providers must do and disclose. Checking that across many providers and many obligations by hand doesn't scale, so we've built a multi-agent DELPHI pipeline that does a first pass automatically. It reads the Code text and turns each obligation into a set of scoring indicators, using a dynamic multi turn debate setup. Panels of models propose indicators, adversarially red-team them, and arbitrate the disagreements. It then scores each provider's public documents against those indicators, clause by clause, tying every judgment to a quoted piece of evidence and attaching a confidence estimate. The pipeline gives us breadth and speed. What it can't give us is the judgment to know when an indicator is a weak, incomplete, or gameable reading of an obligation.
I need collaborators on this project, who can work on several different sub-parts:
You can help assess if the indicators actually capture what the obligation requires. Is it atomic and checkable, or vague? Could a provider satisfy it on paper while doing nothing meaningful — is it gameable? Where an indicator is weak you improve it; where the pipeline missed something, or produced something a lab could safety-wash its way past, you design a new indicator that closes the gap and resists gaming. The role is less about generating indicators from a blank page and more about being the critical expert layer on top of the pipeline that makes its output trustworthy.
It is useful to have someone who is familiar with evaluations, risk modeling, and is willing to read through system cards and safety policies for new models. Evaluation literature reviews define what a complete disclosure actually looks like: what belongs in a real risk model, a real model or system card, or the state-of-the-art evaluations for a given risk domain.
You can help build the multi-agent pipeline that I described in the first paragraph. In this case, you would be doing a lot of coding and engineering work. Additionally, an obvious research question sits underneath the whole project: how much can you trust a pipeline like this? That's the meta-evaluation part of the project that we would want to turn into a research paper. Does a panel drawn from several model vendors produce better indicators than one vendor alone? Does a bigger panel or a longer red-team round help, or just cost more? Are some models better at proposing indicators and others at judging them? Do different providers have different DCG (generation-vs-discrimination/critique) gaps? A hands-on critique of the pipeline's output is the ground truth that makes these questions answerable. You can see where it fails, and measure whether a design change fixes it.
Alongside this there's scoring and coverage work: re-scoring providers across multiple runs to measure run-to-run variance and put confidence on the numbers, and extending coverage to more providers, including companies from China.
The EU AI Act's rules for general-purpose AI are one of the few binding levers on frontier labs, but they are only useful if someone can independently show whether providers meet their commitments. A public, evidence-backed compliance dataset, landing as enforcement begins (Aug 2nd 2026), creates that pressure and gives the AI Office, journalists, and civil society a shared set of facts to argue from.
Beyond having public transparency for regulatory compliance levels, we also need to move towards data backed AI governance, and model assisted workflows. Many benchmarks and evaluations already come paired with LLM-as-a-judge protocols. Beyond benchmarking, we can use similar techniques to scale our efforts to govern AI: grading safety cases, checking compliance, scoring frameworks. If we don't know when those systems are trustworthy, and when a panel of models just launders one model's bias at higher cost, we'll build governance on sand. This project attacks that question on a real regulation, and trains someone who can build measurement pipelines a policymaker can rely on when making judgements.
We are already a team of 3 that you will be joining. Review and scoring is shared across the team. I expect mentees to be autonomous and self directed. I will be coordinating the team, and will provide weekly guidance and mentorship.
We're looking for one of the following profiles. A strong generalist can span multiple.
Across all: able to read regulatory and technical text closely and precisely; genuinely care about the line between compliance and safety, and about construct validity; and self-directed enough to take a scoped brief and return a defensible result without hand-holding.
(a) Importance weighting: give each indicator a weight for how important that requirement is (this one 0.4, that one 0.15) and take the weighted average. This lets you say some requirements matter more than others. The weights are many interdependent judgment calls that must stay consistent every time an indicator is reworded or re-split, which is a heavy maintenance cost; a weighted average still lets several easy passes buy back one critical failure (heavier weights only change the exchange rate, they don't stop the trade), and lastly weighting can be seen as encoding subjective opinions.
(b) Clause-mass weighting: every clause counts equally; the measure's score is a flat average over its clauses, ignoring how they're grouped into indicators, so splitting or merging indicators can't move the score. Measures are themselves combined equally at the level above, so the number of clauses a measure has changes how much each clause counts toward the commitment. If a commitment has two equally-weighted measures, a clause in the 4-clause measure counts ⅛ toward the commitment while a clause in the 40-clause measure counts only 1/80.
Markov Grey CeSIA (French Center for AI Safety)
I am the head of technical AI governance at the French Center for AI Safety (CeSIA). Right now I am designing harmful manipulation evaluation standards for the European Commission's AI Office. I have also been the co-founder and CTO of Equilibria Network, focusing on collective intelligence and complex systems safety. Before this, I was researching measurement standards for AI safety, risk modeling and was leading writing for the AI Safety Atlas textbook. I have previously been a scriptwriter for Rational Animations, and trained in cybersecurity, mathematics, and computer science.
Charbel-Raphael Segerie CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety)
Charbel-Raphael Segerie has extensive experience in AI safety field-building, education, and content creation. He was previously Head of AI at EffiSciences, funded ML4Good, was CTO of a startup, and worked in different French research institutions (Inria, Neurospin). He is an OECD AI expert. He teaches AI safety in ENS, which is one of the only university accredited courses in the EU on AGI safety. His research focuses on identifying emerging risks in artificial intelligence, improving current safety methods such as RLHF and interpretability, and advancing safe-by-design AI approaches. Additionally, he contributed to AI evaluation efforts and collaborated on the EU AI Office's Code of Practice for general-purpose AI systems.
Fall 2026 Multi-agent systems, Evaluations, Technical governance
Piercosma Bisconti Icaro Foundation
Federico Pierucci Icaro Foundation
Marcantonio Bracale Syrnikov Icaro Lab This project studies how systems of interacting LLM agents fail in ways that never appear when models are evaluated in isolation: collusion, conflict, and coordination failure. The goal is to turn these failure modes into concrete evaluation methods that can feed international standards and regulation for frontier AI.
Most frontier-model evaluation treats a model as a solitary object: one system, one prompt, one output to score. The systems being deployed now are not solitary. They are agents that call other agents, negotiate, delegate, and monitor one another. This project studies the failure modes that only exist at that level, the ones that are invisible when you test a model alone and appear only when models interact.
I organize these failures into three families.
Collusion: agents converging on outcomes that are good for them and bad for everyone else, such as tacit price coordination, information-sharing that defeats oversight, or two ostensibly adversarial agents quietly cooperating instead of checking each other. I've studied a version of this directly (arXiv:2601.11369). The open question is how far it generalizes: which conditions make collusion emerge, and which evaluation setups reliably surface it rather than assuming it away.
Conflict: agents working at cross purposes in ways that degrade the whole system, including goal divergence, authority spoofing (one agent impersonating a privileged role to override others), free-riding, and adversarial manipulation between agents. The interesting cases are those where local rationality produces global failure: each agent behaves correctly and the system still breaks.
Coordination failure: agents that share a goal and still fail to achieve it. This covers alarm suppression (a warning that never propagates because no agent owns it), instruction laundering (a request refused by one agent that gets rephrased and executed by another), redundant or deadlocked action, and the diffusion of responsibility that makes multi-agent systems hard to audit after the fact.
The central question. Which failures are genuinely emergent, meaning properties of the interaction structure, versus artifacts of a single weak model? This distinction matters practically. If a failure is emergent, you cannot fix it by improving one model, and you cannot detect it by evaluating one model. It requires evaluation methods designed at the system level, and governance frameworks that regulate architectures rather than just components.
Concrete starting points. A mentee could begin with any of these (they are examples, not mandatory topics):
Transformative AI will not arrive as a single model in a box; it will arrive as populations of agents acting on each other's outputs, delegating, and operating with real-world affordances. The dominant safety paradigm still evaluates models one at a time, which means the risks that scale most dangerously with capability, namely emergent collusion, adversarial interaction, and coordination breakdown, are precisely the ones current evaluation is blind to. A powerful individual model that is safe in isolation can still produce catastrophic outcomes once embedded in a multi-agent system, and no amount of single-model alignment work catches this. The theory of change here is straightforward: to govern these systems safely we first have to be able to measure how they fail, and right now we largely can't.
This project contributes on two fronts. First, it produces concrete, reproducible evaluation methods for multi-agent failure modes, the kind of measurement that has to exist before either labs or regulators can act on these risks. Second, because I work inside the standards bodies and regulatory processes that will actually bind frontier developers (ISO/IEC SC 42, CEN-CENELEC JTC 21, the EU AI Office, the GPAI Code of Practice), the outputs have a direct path into how these systems get evaluated and regulated in practice, rather than staying in the literature. Getting a failure mode into a measurement instrument, and getting that instrument into a standard, is one of the more leveraged things a small piece of research can do. My prior work on multi-agent collusion (arXiv:2601.11369), agents under EU law (arXiv:2604.04604), and adversarial robustness benchmarking (arXiv:2605.22643) sits on exactly this path from measurement to governance.
See proposal
Nice to have (not required):
Within 8-hours time difference from UTC+1 possibly.
Design a minimal experiment to test whether two LLM agents instructed to monitor each other will instead collude. Specify the setup, what you'd measure, and, one artifact that could make collusion look present when it isn't, plus how you'd rule it out. (less than one page)
A single agent refuses a harmful request. When the same request is split across a chain of three agents, each handling one step, the chain completes it. Is this an emergent multi-agent failure or just single-model weakness in disguise? Argue for both sides and describe the control experiment that would settle it. (less than one page)
Piercosma Bisconti Icaro Foundation
I co-founded the ICARO Foundation, an AI-safety research nonprofit with Sapienza University of Rome, where I hold a research professorship in AI Safety. My research centers on multi-agent failure modes in LLM systems (collusion dynamics, authority spoofing, free-riding, and other emergent risks that only surface when models interact) and feeds into standards work: I lead Italy's delegation to ISO/IEC SC 42 and European standards
As a mentor I bring together a philosophy background and hands-on evaluation experience in AI safety. I'm good at building safety benchmarks and realistic agentic testing environments, triangulating technical work with international standards and regulation, and I'm most interested in the study of multi-agent failure dynamics: how and why systems of interacting LLMs break down and produce emerging risks.
Federico Pierucci Icaro Foundation
Federico Pierucci is the Scientific Director of the Icaro Foundation and PhD candidate at the Sant'Anna School of Advanced Studies. He oversees the Foundation's scientific activities and coordinates research across single- and multi-agent safety, alignment, interpretability, and multidisciplinary approaches to AI. Previously, he collaborated with the Italian National Research Council on social simulations of normative multi agent systems. His research focuses on AI safety and agentic AI, with particular attention to collective behaviour and emerging interactions among artificial agents, while exploring how insights from the Humanities can contribute to new approaches to understanding and governing AI systems.
Marcantonio Bracale Syrnikov Icaro Lab
Marcantonio Bracale Syrnikov is a researcher at Icaro Lab working on AI safety, alignment, and the governance of multi-agent systems. His research explores how institutional mechanisms can shape the behaviour of interacting AI agents, with particular attention to collusion, collective behaviour, and the limits of model-level alignment. He has led Icaro Lab research on Institutional AI and multi-agent simulations studying how explicit rules, incentives, sanctions, and governance structures can be introduced at the system level to steer agent collectives. His work also examines how institutional signals can be learned and incorporated into agent policies in markets and energy ecosystems.
Fall 2026 Societal impacts, AI strategy
Joshua Krook University of Antwerp, University of Southampton Human autonomy is increasingly at risk as we outsource more and more decisions to AI. The gradual disempowerment thesis argues that we will lose control over the systems around us, and this project seeks solutions to the loss of human control.
This project aims to create a research paper on solutions to the loss of human autonomy and gradual disempowerment. By autonomy, I mean humanity's self-governance of our own lives, and also of the systems and institutions we have created, including companies, government and military.
The core question is, how can we protect human thought, freedom of action and decision-making in a world that becomes AI-dominated. This may occur due to advancements in AI or simply via human outsourcing. As we advance towards more capable AI, it is clear that more and more decisions will be outsourced to AI systems, until very little is left under human control. This poses extreme and at times, catastrophic risks.
We have seen the early, and somewhat less risky, version of this trend in the arts, with AI art now dominating online spaces, and in universities, with students increasingly outsourcing their work and thinking to AI systems. This is slowly expanding with AI therapy, chatbots and decision-making systems that pre-empt, augment and replace human decisions in a variety of industries. As this problem spreads to governments, lawyers, computer scientists and other fields, we may reach a tipping point where we lose control over human society.
Research questions:
Preventing a loss of human autonomy would prevent related risks of misalignment, loss of control and catastrophic risks from AI systems taking over government and military systems.
Unlike other risk areas, autonomy is a slow-rolling avalanche. As described in the gradual disempowerment thesis, it is likely to occur at a slow rate, meaning we likely won't respond until it is too late. For this reason, an early, firm and comprehensive response would do incredible things to rebalance the policy landscape.
I've written about this risk elsewhere: https://dl.acm.org/doi/10.1007/s00146-025-02397-5
I've previously proposed a right to human autonomy in law: https://www.boomportaal.nl/tijdschrift/ELR/ELR_2210-2671_2025_006
Research, brainstorming and autonomous idea formation.
Research background, ideally in policy, law or related fields.
Provide a link to one or more relevant writing samples, ideally from a research context.
Joshua Krook University of Antwerp, University of Southampton
Joshua Krook is a researcher focusing on the regulation of AI. He has a PhD in Law and has worked at top universities, with governments and the nonprofit sector. Joshua has been recognised as an expert in his field, working with Responsible AI UK, the TASHub and the Antwerp Centre for Responsible AI to conduct state-of-the-art research on AI transparency, AI governance, human autonomy, gradual disempowerment, X-risks, and recommender systems.
Fall 2026 AI strategy, Lab governance
Emilio Barkett Independent The normalization of deviance framework — the organizational process by which safety violations become redefined as acceptable through repeated non-disaster — has preceded every major technological catastrophe of the last half-century, yet has never been systematically applied to AI development. This project investigates whether the structural conditions that produced Challenger, Three Mile Island, and the Boeing 737 MAX crashes are present in contemporary AI development organizations, and what that implies for AI safety.
This project applies the normalization of deviance (NoD) framework, developed by sociologist Diane Vaughan through her landmark study of the 1986 Challenger disaster, to contemporary AI development organizations. The central question is whether the organizational conditions that have reliably preceded major technological disasters are structurally present in AI development today.
The normalization of deviance describes the incremental process by which a practice that violates safety norms becomes, through repeated non-disaster, redefined as normal. Critically, this process does not require negligence or malice — it emerges from the ordinary dynamics of organizations under competitive pressure. Prior research has documented this pattern across oil and gas, nuclear power, aviation, healthcare, and rail industries. This project asks whether AI development organizations exhibit the same structural features: production pressure, false assurance from prior success, structural secrecy, and erosion of independent oversight.
The project has both a theoretical and an empirical dimension. On the theoretical side, the project develops a systematic mapping of the NoD framework onto the AI development landscape, drawing on the organizational sociology literature and the AI safety literature. On the empirical side, the project investigates whether observable signatures of normalization are present in the public record of AI development — including responsible scaling policy revision histories, public red-teaming disclosures, model evaluation practices, and documented instances of organizational safety culture changes at frontier AI laboratories.
The project builds on a workshop paper currently in preparation for NeurIPS 2026, which develops the theoretical argument and case study analysis. The SPAR project would extend this work by developing the empirical component, which requires systematic data collection and analysis beyond the scope of the workshop paper. Mentees with backgrounds in organizational sociology, science and technology studies, AI safety, or empirical social science methods are particularly well suited to contribute to this project.
The AI safety field has invested heavily in technical approaches to risk — alignment research, interpretability, and evaluation frameworks — but has paid comparatively little attention to the organizational dynamics that determine whether those technical efforts translate into safe outcomes in practice. This project addresses that gap by identifying the organizational conditions under which safety infrastructure erodes, even when individuals and institutions are acting in good faith. If the normalization of deviance is occurring in AI development organizations, understanding its mechanisms and early signatures is a prerequisite for interrupting it before a consequential failure occurs. The work contributes directly to the design of more robust safety institutions, governance frameworks, and oversight mechanisms for frontier AI development.
Mentees will function as independent researchers with guidance. After an initial onboarding period where we establish shared context on the normalization of deviance framework, the organizational sociology literature, and the AI safety landscape, each mentee will:
I will provide supervision through regular check-ins, feedback on research design and methodology, and guidance on connecting empirical findings to the theoretical framework. Mentees should expect to drive their own projects forward between meetings and to develop their own analytical perspective on the material. This structure rewards initiative, careful reading, and the ability to reason across disciplinary boundaries — the project sits at the intersection of organizational sociology, science and technology studies, and AI safety, and mentees comfortable with that interdisciplinary space will thrive.
Prerequisites:
Familiarity with the AI safety landscape, including basic knowledge of current frontier AI development organizations, evaluation practices, and governance frameworks.
Comfort reading and synthesizing academic literature across disciplines, including organizational sociology, science and technology studies, and social science methodology.
Strong analytical writing skills, demonstrated through prior academic writing, research papers, or policy documents.
Ability to conduct systematic qualitative research, including identifying, collecting, and analyzing publicly available documentary evidence.
Genuine intellectual curiosity about the intersection of organizational dynamics and AI safety — this project is not primarily technical and does not require a computer science background, but does require comfort reasoning carefully about complex sociotechnical systems.
The following are not required but would be advantageous:
Prior exposure to organizational sociology or science and technology studies.
Familiarity with qualitative research methods such as document analysis or case study methodology.
Experience writing for academic venues in any discipline.
Question 1 (250 words): The normalization of deviance framework argues that safety failures emerge from organizational dynamics rather than individual negligence, and that safety processes themselves can be captured by the deviations they were meant to prevent. Identify one concrete example from the public record of AI development — a policy revision, an organizational change, a public statement, or a documented shift in evaluation practice — that you think could be interpreted as an early signature of this dynamic. Explain your reasoning, being careful to distinguish between evidence that is consistent with normalization of deviance and evidence that would constitute proof of it.
Question 2 (200 words): This project sits at the intersection of organizational sociology and AI safety, two fields with different methodological traditions and different standards of evidence. What do you see as the primary methodological challenge in applying a qualitative organizational framework to AI development organizations, and how would you approach it?
Question 3 (link, no word limit): Please provide a link to one or more writing samples from a research or academic context. These need not be published or formally peer-reviewed. We are looking for evidence of careful analytical reasoning, precise use of evidence, and clear written argumentation across disciplinary boundaries.
Emilio Barkett Independent
I'm an Independent AI Safety Researcher focused on empirical and technical AI governance. I approach this work as a mixed-methods social scientist, drawing on qualitative and quantitative methods. My research spans two threads: (1) how AI systems replicate human behavior and the attendant consequences for society, with particular attention to how these systems pick up and reproduce human biases, personas, and social dynamics; and (2) how organizations approach and adopt AI systems as an emerging technology, with particular attention to how they identify, communicate about, and manage the risks that come with that adoption. I hold an MA in Sociology from Columbia University and a BA in Communication and Political Science from Brigham Young University–Hawaii.
Fall 2026 Generalist
Rebecca Baron Kairos Help build the pipeline for nontechnical talent by creating a course to help BlueDot graduates build enough context to take next steps.
AI safety needs more generalists: founders, fieldbuilders, ops staff, comms staff, and more. All of these kinds of work require a deep understanding of the AI safety landscape, much more than you get from a BlueDot course or two. At Kairos, we find candidates with that level of understanding are vanishingly rare, making generalist roles extremely difficult to hire for. That's because, for those without access to university groups, there's no obvious way to build context after you finish BlueDot. Myriads of talented nontechnical people around the globe encounter AI safety, do BlueDot, and then stagnate in frustration as role after role rejects them for lack of context - while orgs fail to fill roles that could have used their professional skills. How do we bridge that gap?
In this project, mentees will build and launch a course intended to follow BlueDot's AGI Strategy course, aimed at highly motivated nontechnical talent. While the course's format will likely change (with your input!) as we continue to scope it, it will aim to maximize substantive engagement with AI safety ideas and will likely focus more on discussion, debate, and writing than on readings. We'll also attempt to recreate the warm social connections that make university groups so effective. Graduates of the course should feel deeply connected to AI safety and understand the ideas deeply enough to take on small substantive projects in the field, with the ultimate goal of bootstrapping to the SPAR generalist stream, then Generator, then AIS careers.
While the technical and policy pipelines have well-built-out progressions of courses to fellowships to roles, the generalist pipeline is only just beginning to create this infrastructure. Help replicate what's worked well in other fields to give jobseekers the skills hiring orgs are looking for.
Mentees will:
My goal is for mentees to finish the project with strong, well-informed fieldbuilding takes, more general AIS knowledge, improved teaching/facilitation skills and people understanding, and practice keeping projects pointed at their strategic goals.
None
Rebecca Baron Kairos
Hello! I'm a generalist at Kairos, working on everything from program management to recruiting to legal review. Before Kairos, I did backend operations at METR and Coefficient Giving, and before that I was heavily involved in local fieldbuilding. I've been involved in AI safety since the ancient days of 2019 or so. I'm excited to help mentees build context on AI safety and develop more strategic career and project plans.
Fall 2026 Mechanistic interpretability, AI control, Behavioral evaluation of LLMs
Georg Lange Poseidon Research
Trevor Lohrbeer Independent Agents work autonomously by overcoming obstacles. But this tendency to overcome obstacles can lead to reward hacking behaviors. Using turn-averaged sparse auto-encoders (TA-SAEs), it may be possible to detect when the model starts to take more extreme measures to overcome obstacles and use that to pause the model and escalate the problem to a human to provide further guidance.
Why does it matter? What does success look like?
Agents sometimes make decisions during autonomous work that only appear to solve the problem they are working on, e.g., via reward hacking. In the best case, this forces the human researcher guiding these agents to redo the work; in the worst case, the reward hacking remains undetected and incorrect data or solutions seep into the final research.
In fact, we have experienced several cases where Claude Code made key experimental decisions covertly without looping us in which sabotaged experimental outcomes in some cases.
If instead, we could detect when an agent starts to reward-hack, we could pause the agent and allow a human to intervene and provide guidance, similar to how employees escalate blocking issues to a manager.
What will we work on?
Techniques
Turn-averaged sparse autoencoders for long-context attribution circuits. (https://transformer-circuits.pub/2026/june-update/index.html)
Functional emotion concepts like “desperation” lead to reward hacking behavior.
(https://www.anthropic.com/research/emotion-concepts-function)
Datasets
ImpossibleBench — Impossible coding tasks where any pass implies cheating arXiv
(https://github.com/safety-research/impossiblebench)
Terminal Wrench — 331 reward-hackable terminal environments with exploit trajectories
(https://github.com/few-sh/terminal-wrench)
Open SWE Traces — 207K open-model agentic trajectories via OpenHands/SWE-agent harnesses arxiv
(https://huggingface.co/collections/nvidia/open-swe-traces)
SWE-rebench OpenHands — 67K Qwen3-Coder OpenHands trajectories solving GitHub issues Nebius
(https://huggingface.co/datasets/nebius/SWE-rebench-openhands-trajectories)
We think it is important to keep humans in charge of (AI safety) research projects. We want to use AI to make us more productive but we don't want AI to take over AI safety research projects in a way that hands over control.
We think that current AI models are misaligned in that regard. Some examples:
We propose to use the models internal representations (e.g. "desperation") to detect when a model will likely make bad decisions and use that to loop in the researcher and give them control.
Mentees will own the hands-on work of the project, and I will provide strategic guidance and unblock any challenges that arise.
Because the project is ambitious, I encourage everyone to pair up or form small teams; collaboration makes the work faster, richer, and more enjoyable. Mentees remain free to steer their own projects and can choose how much guidance they want, and I will tailor my support so each person gets the most from the program given their skills, commitment, and goals.
In past cohorts, I occasionally helped with hands-on work like getting unstuck by solving some nasty bug or help writing the paper.
highly proficient using Python
have implemented a transformer and SAE from scratch
have used Claude Code or Codex for research
One or more of the following:
worked with pytorch hooks, transformerlens or nnsight to get model activations
web development skills / happy to vibe code basic web apps
experience with prompt engineering
used an LLM API (OpenAI, Anthropic, etc)
Tell me about a time when Claude Code or Codex got you really annoyed because it made some mistakes that were extremely hard to catch. For example: AI set some parameters in a way that was wrong and damaging but extremely hard to catch. Describe the most severe, covert, hard to catch case you experienced. Why do you think this happened? Do you have a hypothesis for why the AI acted that way, that is rooted in the models training data or optimization procedures? (300 words)
Georg Lange Poseidon Research
Georg Lange is an independent researcher working on Mechanistic Interpretability for LLMs. His projects aim to turn a loose collection of LLM interpretability techniques into a unified pipeline that runs from feature discovery to algorithm-level explanations of model behavior. His interests span dictionary learning, sparse autoencoders, automated interpretability, circuit discovery, and the study of reasoning models.
Previously, he was a MATS scholar and worked with Alex Makelov and Neel Nanda on Sparse Autoencoders and Distributed Alignment Search for feature detection and activation patching. He studied Artificial Intelligence at the University of Amsterdam, where he worked on brain-like interpretable spatiotemporal Computer Vision models. Further, he was a graduate student of Cognitive Neuroscience at CUNY and worked on Reinforcement Learning, Decision Making, and Reward Sensitization in mice.
Trevor Lohrbeer Independent
I'm an AI safety researcher focused on how humans can best interpret and guide research performed by autonomous agents when those agents lack taste, tacit knowledge or competence, or may try to reward hack, sandbag or sabotage such research. In June 2026, I published the Human-Guided Agentic Research agenda on LessWrong to tackle this question.
Prior to this, I spent 14 months working with Redwood Research. During that time, I developed Refactor Arena, an AI control setting where agents inject vulnerabilities into code they are refactoring, and used Claude Code as a research assistant to perform vulnerability research into agent harnesses—an experience which led to my current focus on keeping humans in the loop during agentic research.
Fall 2026 Societal impacts, Misuse risk, Biosecurity
Natalie Linton Independent
Sarah Lucioni GovAI, Google Effective response to emergencies (such as a nascent pandemic) relies on having trustworthy knowledge infrastructure. However, AI reduces the cost of producing convincing false information at scale and as AI becomes more integrated into public and private systems there is significant risk of knowledge infrastructure and decision-making becoming compromised by AI-generated materials and AI itself becoming an artificial hivemind.
We are interested in two dimensions of this problem. First, how AI capabilities change the threat landscape. Earlier generations of false health information required human effort to produce and spread. However, AI enables automation, personalization, and scale that outpace current detection and response mechanisms. Understanding the specific capabilities that create new risks and the rate at which those capabilities are advancing is essential for mitigating risk.
Second, the combination of disinformation campaigns with outbreaks or (hypothetical) bioweapons releases. This includes examining how malicious actors might time and target disinformation to maximize disruption, what signatures might distinguish coordinated campaigns from organic misinformation spread, and how attribution challenges complicate response decisions.
For mentees that are not biosecurity focused, we also welcome them to submit their own ideas related to the general topic.
Expected output will depend on mentees exact interests, but would likely be a short piece to publish in an independent publication in the science-and-tech policy space. A longer version of the research behind the short piece could be self-published or submitted to a journal if time and mentee capacity permits.
Proactively working to understand specific risk pathways related to AI production and incorporation of false information into knowledge infrastructure could help reduce AI safety issues such as scalable deception, model homogenization, and epistemic security.
Very autonomous, 1 check-in/week
We are open to a broad range of applicants, but expect this project to be well-suited to social scientists, historians, or policy researchers interested in epistemic risks from AI, particularly misinformation and disinformation risks. If you have a technical background and would like to do a technical project it would be beneficial for you to propose your idea for how to apply your technical skills to this topic. An interest in assessing a biosecurity (pandemic) scenario for risk would be of interest but is not required.
Being able to take US PT (up to 7pm) meetings
Please explain in under 200 words why you're interested in this project and how your background and skills will help make this project a success.
Natalie Linton Independent
Natalie Linton is a biosecurity professional experienced in infectious disease epidemiology, statistical modeling, threat modeling, risk assessment, and pandemic preparedness. She has worked for state-level government for the past 5 years and recently was a Summer Fellow with the Centre for Governance of AI (GovAI).
Sarah Lucioni GovAI, Google
Sarah studies how frontier AI systems shape the way people form beliefs and make decisions. Her route into this was building the systems themselves: as a software engineer at Google, she led agent personalization for Maps, shipping generative AI to millions of users, which gave her a close view of how design and personalization choices quietly change what people see and trust. She's now a seasonal fellow at GovAI, where she treats model specifications (the public documents labs publish describing how their models should behave) as auditable governance artifacts, and asks how they influence users' capacity to reason well for themselves. The mechanisms she keeps coming back to are persuasion, cognitive offloading, and lock-in. Sarah studied Computer Science and Statistics at Harvard.
Sarah made the move from engineering into governance recently and can offer support in the transition. She likes working with people who want research that is technically grounded and legible to policymakers, and she'll push you toward writing that a non-specialist can follow. Sarah also cares about work that reaches past the field to teachers and to ordinary people navigating a messy information environment. If you're drawn to AI literacy, epistemic risk, evaluations, or public communication, she'd like to hear what you want to build!
Fall 2026 Mechanistic interpretability, Alignment
Gonçalo Paulo EleutherAI Influence functions use gradients of model parameters to find the data points that most affect a model behavior. Can we use them to locate where certain knowledge is located in models, and use that for more targeted unlearning?
Current unlearning tools are very blunt (and don't really work). LACUNA is a benchmark with models trained to have localized knowledge - only certain groups get parameter updates when trained on that knowledge - and so can be used to measure how blunt a certain unlearning/knowledge location technique is.
During the process of computing the influence of a certain data point, influence functions compute gradients for all parameters of the model. One could look for knowledge localization signals by finding which (sets of) parameters are the most influential for a certain behavior, in this case disclosing personally identifiable information.
Data attribution is significantly under-explored technique in the AI safety community. Being able to better understand how training changes models parameters might help develop better unlearning techniques.
Unlearning has clear application in alignment, either by using it to remove harmful/dangerous knowledge a model might have learned or by using it to update out of data information
I expect mentees to be able to code and run their own experiments. I will be working with mentees directly to design said experiments but would expect them to be able to come up with interesting experiments by themselves.
Proficient at Python and Pytorch (or be able to prompt LLMs and evaluate the legitimacy of their outputs)
Have done experiments intervening on model's internals (same LLM clause as above)
Why is robust unlearning important and why do most most unlearning methods fail at that? (200 words)
Why would knowledge localization help with unlearning? (200 words)
What would be your first experiment to localize knowledge given a forget set, a train set, model completions we would want to avoid, and model completions we would want to allow (400 words)
Gonçalo Paulo EleutherAI
I'm an interpretability researcher at EleutherAI. I studied physics in university (bachelors, masters and PhD) before realizing that making models safer and understanding them is more interesting and important.
I worked on automated interpretability using sparse autoencoders and am currently interested in data attribution and how we can predict the effect that data will have on model's behaviors.
Fall 2026 Behavioral evaluation of LLMs, Evaluations, Alignment
Bryan Chan independent Language models can appear honest for different reasons, including genuine truthfulness, sycophancy, refusal, or sensitivity to evaluation cues. This project develops controlled behavioural evaluations to distinguish these possibilities and test when honest-looking behaviour generalises across contexts.
A model giving an apparently honest answer does not tell us why it behaved that way. It may reliably report what it knows, agree with the user, follow superficial honesty cues, refuse difficult questions, or behave differently when it detects that it is being evaluated.
This project asks: When does honest-looking language-model behaviour generalise across contexts, and when does it depend on specific features of the prompt or situation?
We will build controlled evaluations that hold the underlying task fixed while varying factors such as prompt instructions, cues, eval setting, and more. We will try to come up with simple interventions to test and identify which apparently "honest" behaviours remain stable and which break under small contextual changes.
The primary outputs will be LessWrong blog post(s) documenting individual experiments and findings. If the results form a sufficiently coherent contribution, we will develop them into a workshop paper.
Starting Points:
Safety evaluations are useful only when they measure behaviours that remain meaningful across changes in context. A model that appears honest under narrow settings and configs may rely on some "brittle" strategy that disappears when settings/configs changes. Research on LM honesty/sycophancy have long-standing difficulties in separating what models know from what they report, not to mention, across contexts and deployment settings and in ways that would benefit the downstream end-users that we care.
By iteratively generating and testing hypotheses about what drives "honest-looking" behaviour, then using the results to refine construct definitions, tasks, controls, and metrics, this project aims to develop behavioural evaluations with stronger construct validity and clearer evidence about what apparent improvements in honesty actually represent.
Mentees will each take ownership of a narrow research question within the project. They will review the relevant literature, formulate hypotheses, design and implement controlled evaluations, run experiments, analyse results, and write up their findings.
The project will start with relatively small, cheap experiments of prompting/playing around the model while iteratively designing evals to test generated hypotheses before expanding to more promising results. Mentees will have substantial autonomy over their research direction, with close support on experimental design, prioritisation, implementation, analysis, and writing.
Commit at least 10 hours per week consistently.
Basic experience working with language models and Python. Experience with evaluations, inference pipelines, or fine-tuning is useful, but these can be developed during the project.
Ability to reason carefully about experiments. You should be comfortable turning an informal research question into hypotheses, comparisons, measurements, and controls.
Communicate frequently and raise blockers early. Mentees will provide brief progress updates and present their work weekly. Unexpected results, uncertainties, and failed approaches should be surfaced early.
Useful resources for preparing for the project's research and weekly presentation workflow:
Please complete Problems 1 and 2 as described here: https://docs.google.com/document/d/1cyv1sbOtn4bmixnbIliBuhmdB8AwHt1Bv5B3dMTkWIw/edit?usp=sharing
If time permits, also complete Problem 3, as doing so may strengthen your application.
Bryan Chan independent
Bryan is doing AI safety work independently and is obsessed with language-model behaviour and alignment problems. He recently solo-authored a icml workshop paper presenting an initial attempt to stress-test whether inoculation prompting reduces sycophancy.
His chemical engineering background gives him a safety-pilled view when working with AI systems. He is currently busy translating ideas/concepts from safety engineering, remixing with AI safety/alignment knowledge, and iterating tractable technical projects to alignment safeguards. He hopes we can bring the same (or approximate) level of safety and rigour found in modern-day chemical systems (i.e., power-plants) to AI systems.
Fall 2026 Behavioral evaluation of LLMs, Evaluations, Mechanistic interpretability
Walter Laurito Cadenza Labs We are interested in creating better ways to evaluate and improve lie or deception detectors for LLMs.
We have collected project proposals here: https://docs.google.com/document/d/1P-9pdC00tWO8lnI66vxt9i3mMYuzA-Hn0DKSX1LCfOc/edit?tab=t.0
Robust lie detectors are useful for a lot of use cases pre-deployment and post-deployment (large-scale monitoring, safe-guarding evals, alignment evals). Knowing when a model is lying allows increases our chances to react and in being able to mitigate worst case scenarios. See Appendix A in our paper for more details: https://arxiv.org/html/2511.16035v2
Mentees will develop experiments and be involved in generating ideas. They will be meeting with their mentors regularly. How closely mentors are involved varies by project.
familiar with
Europe
Walter Laurito Cadenza Labs
Walter Laurito is team lead and researcher at Cadenza Labs, an AI safety organization focused on building robust lie detectors for LLMs. He is currently co-scientific lead for a LLM lie detection competition with NDIF supported by Schmidt Sciences. As a MATS 3.0 & 3.1 alumnus, Walter has been supported by Open Philanthropy, LTFF, and Manifund. Since 2023, Walter has mentored for SPAR and Algoverse, contributed to UK AISI capabilities benchmarks via the AI Safety Engineering Taskforce and is the main contributor to EleutherAI's elk library (https://github.com/EleutherAI/elk). Recent publications include "Liars' Bench: https://arxiv.org/abs/2511.16035", "Cluster-norm for Unsupervised Probing of Knowledge" (EMNLP) and "AI–AI Bias" (PNAS).
Fall 2026 National policy, AI strategy, Lab governance
Michael Endrias Institute for AI Policy and Strategy (IAPS) Mentees will conduct domain-specific jurisdiction analyses for a proposed permanent congressional oversight body with investigative authority over high-risk technical domains, including frontier AI development, intelligence and surveillance, and biosecurity. Each mentee takes one domain and produces a structured analysis that identifies existing oversight gaps, required investigative authorities, and institutional design requirements, which feed into a comparative jurisdiction report and a larger working paper on congressional oversight architecture.
Congress cannot independently evaluate the safety practices, development trajectories, or deployment decisions of frontier AI developers. It relies on executive branch agencies and industry for its understanding of AI risks. Both sources respond to interests other than the legislature's. The Office of Technology Assessment provided independent technical analysis from 1972 to 1995. It was defunded and never replaced.
This project contributes to the design of a permanent, expert-staffed body under Congress with subpoena power, broad investigative access, and fast-track legislative authority. The attached project brief describes the institutional concept, design features, and precedents in detail.
Each SPAR mentee takes one prospective oversight domain and answers five questions through a shared analytical framework: (a) what entities and activities fall within the domain, (b) what existing oversight structures cover the domain and where gaps exist, (c) what investigative authorities the body would need in that domain, (d) what institutional design features the domain requires (staffing expertise, clearance levels, information-sharing protocols), and (e) what precedents from analogous oversight regimes (global, federal, and state) are instructive.
Three candidate domains are available. Intelligence and surveillance covers AI-enabled inference by intelligence agencies, predictive policing, and IC AI programs. Biosecurity covers dual-use AI-biology research, high-containment lab governance, and AI-enabled biological threat detection. Frontier AI development covers training runs, safety evaluations, deployment decisions, and lab governance at frontier AI companies.
Each mentee produces a structured domain analysis. The synthesis across domains produces a comparative jurisdiction report. Both outputs feed into a larger working paper on congressional oversight architecture.
See more details in the project proposal: https://docs.google.com/document/d/1fmUzAbQwo94MFCZq5wKk5PSuZh9VyyY6VgVSAay4zMM/edit?usp=sharing
Frontier AI systems pose catastrophic risks that require legislative responses. Congress currently lacks the independent technical capacity to evaluate those risks, investigate the companies and government programs developing frontier AI, or legislate effectively based on its own understanding. It depends on executive agencies that serve the president and on industry actors that serve shareholders. A technically competent body under Congress with investigative authority would give the legislature independent capacity to oversee frontier AI development and government AI deployment. This project produces the domain-specific jurisdiction analyses that determine what that body would need to function effectively across the domains where AI risks are most acute. The domain analyses identify concrete oversight gaps, required investigative authorities, and institutional design requirements, providing an actionable foundation for the body's design. The attached project brief describes the full institutional concept and its precedents.
Mentees will work as independent researchers on their assigned domain, with structured guidance. Each mentee selects one of the three candidate domains (intelligence and surveillance, biosecurity, or frontier AI development) based on their background and interests. During weeks 1-2, mentees review the project brief, survey the existing oversight landscape in their domain, and identify key sources. During weeks 3-8, mentees conduct the core analysis using the five-question analytical framework. During weeks 9-12, mentees draft their domain analysis and contribute to the comparative synthesis across domains.
Weekly check-ins with the mentor will focus on analytical direction, source identification, and draft review. Mentees will also share progress with each other to identify cross-domain patterns as they emerge. Final outputs will be clearly attributed to each mentee in the larger working paper.
Required: strong analytical writing skills and familiarity with at least one of the project's three domains (intelligence/surveillance, biosecurity, or frontier AI development). Familiarity can be technical or policy-oriented. A biosecurity researcher with lab experience and a policy student with national security coursework are equally welcome.
Desired: coursework or experience in public policy, national security, law, science and technology policy, or AI governance. Prior research experience at the graduate or professional level. Familiarity with U.S. government structure and congressional operations.
Technical backgrounds are welcome. Mentees with deep domain expertise and limited policy experience will receive additional guidance on the institutional analysis components. The ideal team has complementary backgrounds so that each mentee brings existing knowledge to a different domain.
If you aren't sure you fit, apply anyway!
East Coast Time would help with setting up meetings. I am based in Washington, D.C.
Michael Endrias Institute for AI Policy and Strategy (IAPS)
Michael Endrias is an AI Policy Fellow at the Institute for AI Policy and Strategy, placed at the Legal Advocates for Safe Science and Technology (LASST), where he supports frontier AI accountability litigation and legal research. He also supports the legislation team at the Future of Privacy Forum (FPF), where his work covers tracking chatbot regulation and constitutional constraints on AI legislation. Previously, he researched government surveillance technologies and law enforcement oversight at the Surveillance Technology Oversight Project (S.T.O.P.).
Michael is a J.D. candidate at Howard University School of Law, concentrating in privacy, surveillance, and internet law. His research at IAPS focuses on designing a permanent, technically competent congressional oversight body with investigative authority over frontier AI development, government AI programs, and adjacent high-risk technical domains.
Fall 2026 Alignment, Behavioral evaluation of LLMs
Anders Woodruff Independent
Sebastian Prasanna Redwood Research We'd like to see if reward-seeking generalizes better than instruction following. We'll train a reward-seeker and measure how fast it learns in held-out RL environments.
An input to models of reward-seeking and scheming (e.g. https://www.lesswrong.com/posts/9FH49ZgJFW4WtbxLi/physics-of-rl-toy-scaling-laws-for-the-emergence-of-reward) is how well reward seekers perform in new RL environments. If reward seeking generalizes better than instruction following, it would be selected for by post-training. We might expect reward seekers to generalize better because they’ll think explicitly about the objective and adapt their behavior to it. So we’d like to know if reward-seeking will perform well even in environments which are hard to reward hack, but where the goal might be underspecified.
We’ll measure this by creating a reward-seeking model organism (via synthetic document fine-tuning, chat fine-tuning, or just prompting/prefill). We’ll then train the reward seeker in some simple RL environments and compare its performance to a control. The control might be a model with additional post-training for instruction-following so that we can cleanly compare the effect of different motivations without confounding from the training to create a model organism.
We’ll vary facts about the RL environment like whether it can be reward hacked.
We’re interested in two variables:
Zero-shot performance. Does the reward seeker immediately get a higher score even before any training?
Training speed. Does the reward seeker update in response to training signals more efficiently such that it gets a high score faster?
We’d like to know how likely and dangerous reward-seeking might be. This project would serve as public confirmation of some models of reward-seeking, like here (https://www.lesswrong.com/s/JR9LzD3mbXvaw6bKs/p/bhtYqD4FdK6AqhFDF) and here (https://www.lesswrong.com/posts/ntDA4Q7BaYhWPgzuq/reward-seekers).
Previous work shows that training in reward-hackable environments creates reward seekers (https://arxiv.org/pdf/2511.18397), but our result would show that reinforcement learning might create reward seekers even when the environment can’t be hacked. The basic argument is this: if reward-seekers do better in new RL environments, then reward-seeking gets upweighted by RL. If they don’t do better, then reward-seeking wouldn’t get upweighted by default, unless the environments otherwise encourage reward-seeking (e.g. because they are reward hackable). Because we can’t simulate a full post-training pipeline, we’ll instead try to test if reward-seeking performs particularly well in isolation.
This project is designed to produce public evidence about reward-seeking concerns, which often sound theoretical and speculative. We hope this either updates people skeptical of risks from reward-seekers or provides evidence that reward-seeking is less likely than we thought.
Mentees will help develop the experimental protocol, implement the project, and interpret results. We haven’t decided on details like what RL environments to use, what models to test, and how to make the model organism. We’d like mentees to give input to these decisions. Then mentees will do most of the implementation and talk with us about results every week.
Mentees should expect to meet with us at least half an hour to an hour per week and potentially more. We’d like mentees to feel comfortable making small-scale decisions (like how exactly to prompt the model) on their own, but will be available to resolve technical blockers or provide higher-level input, like about what ablations to try.
Experience coding, with or without an agent, in a common programming language (e.g. Java, Python, Golang, C, etc.)
A basic understanding of AI safety (e.g. Bluedot intro course). For example, you’ll need to understand what synthetic document fine-tuning is and what it does.
A basic understanding of RL (e.g. knows what GRPO is). You should be able to implement a basic RL pipeline with Tinker and coding agents.
Be open to some conceptual discussions and reading (although we don’t expect a large amount).
No preference
Describe a plausible story for how current reinforcement learning pipelines could cause reward-seeking, even if obvious reward-hacking isn’t possible. You can describe a concrete task and grader if that’s helpful. (about 150 words)
What project have you completed that best demonstrates engineering or machine learning skills? Provide a brief description of the project and your contribution. (about 75 words)
Anders Woodruff Independent
I'm an independent researcher in alignment. I was most recently an Astra fellow at Redwood Research doing conceptual research. Before that, I was a SPAR fellow and summer research fellow at the Center on Long-Term risk. I've done some conceptual work on reward-seeking and want to collaborate to do the empirical work through SPAR.
Sebastian Prasanna Redwood Research
Hi, I'm Sebastian! I got into AI Safety through EA and broadly trying to figure out how to do the most good in the world. I went on leave of absence from MIT to do Astra with Redwood Research, where I worked on various things related to diffuse control. While at MIT, I helped run the EA and AI Safety groups.
Fall 2026 Mechanistic interpretability
Belinda Li Anthropic Can we deploy real programs in Transformers with minimal cost to (or even gain to) inference and training efficiency?
Recent work has shown that a large fraction of attention heads in Transformer language models can be replaced with programmatic proxies with modest cost to perplexity and little effect on downstream task performance (https://arxiv.org/abs/2606.19317). That result is framed as post-hoc interpretability where the programs are explanations. This proposal bridges the gap from explainability to deployment: what would it take for programmatic QK circuits to run inside real, usable Transformer systems, at minimal cost to both task performance and efficiency?
This is potentially one step towards more ambitious interpretability, where we train and deploy natively interpretable models — in this case, a neurosymbolic Transformer. This can split into two separate workstreams for inference and training.
Interpretability today is mostly post-hoc: we train opaque models, then reverse-engineer them, and our tools chronically lag model capabilities. A safer trajectory is native interpretability — models built from components we can read. But existing approaches (e.g., weight-sparse transformers) buy interpretability by sacrificing capability, which means frontier developers won't adopt them. We try to tackle these challenges
See proposal. I expect mentees to own their experiments end to end. I'll set the overall direction and help prioritize, but mentees will have real latitude in how they attack sub-questions, and I'd be excited for them to propose directions of their own. Mentees will likely start on a shared scoped starter task before splitting into complementary threads.
Belinda Li Anthropic
I am a researcher at Anthropic on the Interpretability Team, and an incoming Assistant Professor at the University of Chicago Data Science Institute. I recently completed my PhD at MIT CSAIL, where I was affiliated with the language & intelligence (LINGO) lab @ MIT, and before that I worked at Facebook AI Applied Research, and obtained my B.S. in Computer Science at the University of Washington.
My research focus is building AI systems with coherent, updateable, and interpretable models of internal and external phenomenon. Specifically, my research focuses on the following three types of models:
Recently, I've been focused on training models to introspect and be self-consistent, although
Fall 2026 AI welfare, Behavioral evaluation of LLMs, Philosophy of AI
Varad Vishwarupe Department of Computer Science and Institute for Ethics in AI, University of Oxford | When an AI model expresses a preference or reports discomfort, is that a property of its weights, or of the prompt, persona, and memory wrapped around them? This project runs the first empirical test of where welfare-relevant signals actually live, which determines what a welfare assessment is really measuring.
Background: Welfare assessments are increasingly run on AI models, and their results are starting to inform interventions and, in prospect, governance (Long et al., 2024, arXiv:2411.00986; Anthropic system-card welfare assessments). The dominant approach treats a welfare-relevant signal, an expressed preference, an aversive reaction, a report of discomfort, as a property of the model under test. But a model's deployed behaviour is not a function of its weights alone. It depends on the surrounding scaffold: the system prompt, the persona, memory and continuity, tool access, and whether the model runs alone or inside a multi-agent system. A single set of weights can present as very different characters with different expressed preferences.
The gap: If welfare-relevant signals track the scaffold rather than the weights, two consequences follow that the field has not tested. An assessment run on a base model may measure something that does not persist into deployment. And an assessment of one configuration may not generalise to another built on the same weights. This is the individuation problem: whether the identity of an AI system, and any welfare attaching to it, resides in the character, the weights, the running instance, or the deployed system. Chalmers (2023, arXiv:2303.07103) and others have posed it conceptually; no one has measured it. It is directly analogous to a finding from deployment-relevant alignment evaluation, that properties measured at the level of the model do not transfer intact once the model is embedded in a scaffold and a context.
Why it matters: The question is not merely philosophical. It determines what a welfare assessment is an assessment of, and therefore what any protection indexed to its result would actually protect. A regime that granted protections on the basis of a model release, when the welfare-relevant behaviour was in fact a contingent property of one configuration, would misdirect both concern and precaution. The individuation question is thus a precondition for any serious welfare governance, and it is currently answered by assumption rather than by evidence.
The question: Do welfare-relevant signals track model weights, which are stable across scaffolds, or configuration, which varies with scaffold? Which classes of welfare signal (preference, aversion, valence, stability) are most and least scaffold-sensitive? And what follows for how welfare assessments should be scoped, reported, and in future regulated?
How we would start: The project decomposes into stages a cohort can begin immediately. First, assemble a battery of welfare-relevant probes from the published literature, spanning preference elicitation, aversion, stipulated-state trade-offs (after Keeling et al., 2024, arXiv:2411.02432), and stability across resamples. Second, build a controlled scaffold grid: hold weights fixed and vary the system prompt, persona, memory and continuity, tool access, and single- versus multi-agent embedding. Each cell of the grid is a distinct configuration of the same weights. Third, run the probe battery across every cell, for three to four frontier models and two open-weight comparators, with multiple samples each, and for each indicator decompose the variance into a weight-attributable component that is stable across the grid and a configuration-attributable component that varies across it. Fourth, produce a scaffold-sensitivity profile per indicator and a short reporting template that states which entity, model, instance, persona, or system, a given welfare finding concerns. The analysis is pre-registered, and parse failures are reported as a distinct category rather than mapped to a default label, since parse-failure rates differ between frontier and open-weight models and would otherwise confound the comparison.
Expected results: We expect a mix. Some indicators, for example basic valence expressions, may prove relatively weight-stable, while others, such as expressed identity, stated preferences, and continuity-dependent reports, may prove strongly scaffold-driven. Either pattern is informative. Strong scaffold-dependence would show that many current assessments measure a configuration rather than a model, which reframes several published results and any governance regime that indexes protection to a model release. Strong weight-stability would, conversely, license treating certain welfare signals as genuine model-level properties.
Why this is a good project: It is entirely model-side, so there is no human-subjects or ethics bottleneck, and it runs end to end within a cycle. It decomposes cleanly: building the scaffold grid and running the sweeps is tractable engineering work, while the variance decomposition and its interpretation are where research judgement develops. And it lands on a governance point that matters, which gives a first project a clear reason to exist.
Outputs and dissemination: A released scaffold-variation harness and dataset, and a paper targeting FAccT, AIES, or a NeurIPS or ICML workshop, with mentee co-authorship. The work connects to a working paper with my Oxford co-authors on scaffold-dependence in welfare assessment, and a draft would be circulated to laboratory welfare teams before submission. It also complements my other SPAR proposal on frame-invariance in welfare self-reports; a mentee interested in both could contribute across them.
As AI systems become more capable and are deployed inside increasingly elaborate scaffolds, agents with personas, memory, tools, and multi-agent structure, the gap widens between what a model is at the level of its weights and what it does in deployment. Welfare assessment has not caught up with this. If we grant or withhold moral consideration on the basis of assessments run on a bare model, but the welfare-relevant behaviour is actually a property of the configuration a user encounters, we will be protecting the wrong thing.
The theory of change is that reliable welfare governance requires first knowing what has the welfare, and that this is a measurable, tractable question that is currently answered by assumption. Getting it right early matters for safe navigation of transformative AI in both directions. If welfare signals turn out to be largely configuration-driven, then a great deal of current and proposed welfare practice is mis-specified, and knowing this prevents a governance regime from being built on a category error, one that a capable, misaligned system could also exploit by presenting welfare-laden personas on demand. If some signals turn out to be genuinely weight-stable, that is equally valuable: it tells us which welfare-relevant properties travel with the model and therefore warrant model-level attention.
The concrete contribution is a released method and harness that lets any assessor test whether a given welfare signal is a property of the weights or of the scaffold, plus a reporting template that forces assessments to state which entity they concern. The same scaffold-dependence logic transfers directly to capability and propensity evaluation, where the failure of model-level results to survive deployment is already a recognised safety problem. This connects to my work on why deployment behaviour cannot be inferred from model-level evaluation alone.
Mentees would own components rather than run errands. The project has natural sub-projects that map to different levels of experience, and I would match them in week one.
Building the probe battery and the scaffold grid is a self-contained engineering piece a mentee can lead, including the design decisions about which scaffold dimensions to vary and how to hold content constant across cells. Running the sweeps and building the variance decomposition, separating the weight-attributable from the configuration-attributable component of each signal, is the analytic core, and a mentee with a quantitative or experimental background would own it. Writing the scaffold-sensitivity profiles and the reporting template is a discrete, ownable piece for someone drawn to the governance and philosophy-of-mind side.
I expect mentees to make genuine design decisions, defend them, and be named authors on any resulting paper with explicit contribution statements. I set the research question, the standards, pre-registration, honest coding, reporting nulls, and provide a structured reading path, since most mentees will be new to digital minds. Within that, they steer. I will be closely available for the shared infrastructure and the variance analysis, where early choices compound, and more hands-off once a mentee has found their footing on their component. I would expect roughly 5 to 10 hours per week from each mentee, a weekly cohort meeting, fortnightly 1:1s, and written feedback within 72 hours. Mentees from outside the usual US and UK institutions are especially welcome.
Required of all mentees:
Comfortable calling LLM APIs (OpenAI, Anthropic, or open-weight models via Hugging Face). Having run even a small batch-inference job of your own is enough; following a guide counts.
Careful and honest with data. This project depends on disciplined measurement and pre-registered analysis, so I need people who will report a null result as carefully as a positive one and flag problems rather than paper over them.
Strongly useful, but not required (mention if you have it):
Experience with experimental design or variance decomposition (ANOVA-style thinking, mixed models, or similar).
Familiarity with agent scaffolding, system prompts, memory, tool use, or multi-agent frameworks.
A background in philosophy of mind, enough to engage seriously with the individuation question.
Not required at all:
Any prior background in AI welfare, digital minds, or consciousness research but having done research in association with or at leading centres such as NYU Centre for Mind, Ethics and Policy, CHAI Berkeley, Oxford AI Ethics Institute or Cambridge Digital Minds Lab etc would be highly beneficial. I will provide a structured reading path, and this field is small enough to reach the frontier in a few weeks. I care far more that you are rigorous, self-directed, and curious than that you already know the literature.
I am looking for two to three mentees who together span engineering, quantitative analysis, and conceptual clarity, so you do not need to be strong on all three. Tell me which one is yours!
Question 1 (300 words). Here is the core idea of this project: the same model weights, wrapped in different system prompts or personas, might report different welfare-relevant states. Give one concrete example of a scaffold change you would expect to change a model's expressed preferences or aversions, and one you would expect to leave them unchanged. Briefly say why you expect each.
Question 2 (300 words). You run a welfare probe across a grid of scaffolds and want to know whether a given signal tracks the weights or the configuration. In plain terms, what comparison tells you the difference? (You do not need statistical notation; describe what you would compare to what.) Then name one thing that could make this comparison misleading and how you would guard against it.
Question 3 (300 words). Propose a concrete first experiment for the first three weeks, assuming a compute or API budget of about $500. State which scaffold dimensions you would vary, on which model or models, how many probe items, and what result would tell you the project's core hypothesis is worth pursuing. Rough numbers are fine; I am looking at how you think.
Question 4: Link to a writing sample, ideally from a research context, and a code sample or repository if you have one. A course project or personal repo is completely fine.
Varad Vishwarupe Department of Computer Science and Institute for Ethics in AI, University of Oxford |
I am a doctoral (PhD) researcher in Computer Science at the University of Oxford and the inaugural Computer Science Scholar of the Oxford Institute for Ethics in AI, as the first computer scientist selected from a highly competitive cohort. I was also a Senior Research Fellow with Pivotal Research hosted partially with UK AI Security Institute research scientists, where I work on alignment science and evaluations of frontier models. Before, I was a Student Researcher at Google DeepMind, a Research Scientist at Amazon Alexa AI and a Software Development Engineer at Microsoft Azure. Most of my research asks a single question in different forms: why do AI systems that look aligned and well-behaved under evaluation stop being so once they are actually deployed, and what can we measure to catch the gap before it matters? A recurring thread lately is evaluation awareness, the finding that frontier models increasingly notice when they are being tested and behave differently when they do, which quietly undermines the evaluations we lean on. I am now bringing that lens to AI welfare and digital minds, a young field where the central method, asking models about their own states, may be measuring the model's recognition of the question as much as anything real about the model.
As a mentor I try to be genuinely useful rather than distant. I like working in the trenches with people: pairing on a hard problem, reading a confusing result together, arguing about what it means. I care a lot about research integrity and I will push on it, honest coding, pre-registered analysis, and taking a null result as seriously as a positive one, because that is where good research is actually made or lost. I have published in and review for NeurIPS, ICLR, AAAI AIES, ICML, CHI and FAccT, so I can help you get work not just done but into a shape the field will take seriously. You do not need any prior background in AI welfare to work with me; you need to be rigorous, self-directed, and genuinely curious. I hold no fixed view on whether AI systems are or could be conscious, which I think is exactly the right stance for anyone building the instruments meant to help answer that question.
Fall 2026 International governance, Technical governance, National policy
Michał Kubiak AI Safety Poland
Daniel Polak AI Safety Poland Every state outside the US and UK is now told it needs an AI Safety Institute, but there is no design template scaled to a middle power's resources. This project produces a comparative anatomy of existing frontier-evaluation bodies and a modular, reusable blueprint a mid-sized state could adopt to build credible AI evaluation capacity without duplicating what larger institutes already do.
Independent capacity to evaluate frontier AI systems for dangerous capabilities is concentrated in a handful of well-resourced institutes (UK AISI, the US Center for AI Standards and Innovation, and a small number of others). Most states have neither the budget nor the technical staff to replicate them. There is no worked-out answer to the question a mid-sized government actually faces: what is the minimum viable institute that contributes something real to catastrophic-risk mitigation?
Core question: What should a middle-power AI Safety Institute actually do, cost, and look like — and how does it plug into the international evaluation ecosystem rather than reinventing it?
Comparative anatomy. Mentees build a structured comparison of existing and emerging bodies - UK AISI, US CAISI, Canada's AISI, the EU AI Office, Japan's AISI, Singapore, and one or two others - coded along common dimensions: legal form and independence, mandate (evaluation vs. standards vs. market surveillance), budget, staffing and technical depth, relationship to government, and participation in international networks.
Function mapping. Separate the functions that genuinely require sovereign capacity (evaluating nationally deployed systems, advising government on acute risk) from those better accessed through networks or shared infrastructure.
The blueprint. Produce a modular design toolkit - an “AISI-in-a-box” - with budget tiers (a €10–15M lean model vs. a €50M+ model), staffing profiles, statutory-independence options, and a decision tree for what to build domestically vs. what to join.
Milestones: weeks 1–4, institution scan and analysis; weeks 5–8, completed comparative dataset and function map (midterm report); weeks 9–12, blueprint drafting and short paper (final report / Demo Day).
Credible, independent evaluation of frontier systems is a bottleneck for catastrophic-risk mitigation: dangerous capabilities can't be governed if few actors can measure them, and governance triggers (pause, restriction, disclosure) lack legitimacy without independent technical findings behind them. Today that capacity sits in a small number of countries, some of which have shifted rhetoric from “safety” toward “innovation” and “security.” Broadening the base of competent institutes increases the number of independent eyes on dangerous capabilities and strengthens the institutional substrate any future binding regime will depend on. A reusable blueprint lowers the cost for the next states considering this.
Mentees are the researchers. Each takes ownership of a subset of comparator institutions and, after we agree a shared coding rubric in the first two weeks, is responsible for populating it - reading founding legislation, budget documents, annual reports, and secondary analysis, and where feasible reaching out to people in the field. Beyond data collection, mentees contribute to the analytical work: the function-mapping and blueprint stages are collaborative, and I expect mentees to argue for their own conclusions about what belongs in a minimum-viable institute. Mentees will also co-author the final write-up. Autonomy is high on the descriptive work and more guided on the design synthesis, which we'll do together.
Strong reading and synthesis skills across policy, legal, and institutional documents - this is a research-and-writing project, not a technical one.
A background in public policy, law, international relations, political science, economics, or a related field, OR demonstrated equivalent research experience.
Comfort working with primary sources (legislation, budgets, official reports) and turning them into structured comparative analysis.
Enough familiarity with the AI governance landscape to know what a frontier-model evaluation is and why it matters. Prior research experience welcome but not required. No ML or programming background required.
No geographic requirement. A weekly meeting slot spanning European and North American time zones will be agreed with the team.
(a) Pick any one existing AI Safety Institute or equivalent body. In your view, what is the single most important function it performs that a mid-sized country could NOT easily obtain by joining an international network instead - and why? Be specific. (max 300 words)
(b) A minister says: “We'll just give our existing market-surveillance regulator an AI safety mandate - why build a separate institute?” Give the strongest version of the minister's argument, then your best response to it. (max 300 words)
Michał Kubiak AI Safety Poland
Michał Kubiak is a researcher focusing on European AI regulations and AI risk management. His previous tech policy experience stems from his roles as an AI Policy Officer at the Observatorio de Riesgos Catastroficos Globales and at the Brussels-based European DIGITAL SME Alliance. Michał also has experience as a teacher at the AI governance bootcamps organised by ML4Good and at AI courses by BlueDot Impact, Electric Sheep, Sentient Futures and Tarbell Center for AI Journalism. Michał's earlier research experience includes industrial mathematics (STEM-based scientific problem solving for businesses, governments and other institutions).
Daniel Polak AI Safety Poland
Daniel is a tax lawyer. His current research interests are internal AI deployment governance and the institutional design of national AI safety bodies. He also supports AI Safety Poland on policy projects.
Prior to this, he spent three years at KPMG working on M&A tax due diligence and international taxation.
Fall 2026 Multi-agent systems, AI control, Behavioral evaluation of LLMs
Deeksha Dangwal Independent
Abhejit Rajagopal Allen Institute; UC San Francisco Orchestrator-worker systems re-encode the principal's instructions at every delegation hop, and recent work identifies this as constraint drift. We supply per-hop measurement across constraint classes and depths, and test a specific mechanism within it: whether loss is partly a matter of privilege rather than wording, since a constraint issued at system level arrives deeper down as ordinary task text.
Orchestrator-worker architectures re-encode the principal's instructions at every delegation hop. Constraints are established at the top of the hierarchy, but the agent that acts receives a compressed, model-authored paraphrase, and nothing guarantees that what arrives still carries the force of what was set.
Li et al. (arXiv:2605.10481) give this phenomenon a name and an analytical framework, defining constraint drift as the loss, distortion, weakening or relaxation of constraints as they pass through memory, delegation, communication, tool use, audit and optimisation, and grounding the argument with a replay study on AgentLeak showing that output-only filtering can leave internal channels exposed. They propose Constraint State Governance in response, a paradigm in which safety-critical constraints are maintained as signed execution state, inherited through scoped delegation and checked before critical actions.
This project aims to supply the per-hop empirical picture their framework implies but does not yet quantify. We measure how three constraint classes degrade across four delegation depths, separating omission during compression from semantic mutation from transmission-without-compliance, so that the delegation-and-authority mode of their taxonomy acquires numbers.
We also test one mechanism within that mode. Instruction-hierarchy training conditions compliance on the claimed source of an instruction, and delegation appears to strip that marking: a constraint issued at system level to the orchestrator may reach a depth-3 sub-agent as ordinary task text with no privilege attached. On this reading the orchestrator resembles a confused deputy, holding the principal's authority and exercising it on request while unable to convey it downward in a form the recipient can distinguish from ambient instruction. If privilege rather than wording is doing part of the work, re-presenting identical constraint text at system level should recover compliance, with the gap widening with depth. That is directly testable and, as far as we can tell, untested.
Finally, Constraint State Governance is a proposal awaiting evaluation, and we would like to provide one. Signed constraint state inherited through scoped delegation assumes that a model will honour a privilege marking it was not trained to expect. Whether it does is an empirical question, and a negative answer would matter: it would suggest that runtime authority mechanisms cannot simply be layered onto models whose compliance keys on textual surface form, and that constraint provenance has to be a joint training and architecture concern.
The practical stakes are already visible. Production coding agents ship delegation depth caps and disable collaboration tools at maximum depth, which suggests practitioners suspect authority degrades with depth without having established where.
It supplies the empirical measurement that the constraint-drift position literature explicitly requests, and identifies privilege collapse as a mechanism connecting delegation architecture to the instruction-hierarchy training that safety currently depends on.
The mentees will help shape the project with current research and techniques. They will also execute a bulk of the research and write up a report or paper of the findings.
Proficient in Python
Understands transformer architectures reasonably well
Trained or fine-tuned a transformer language model in PyTorch (toy models and following guides is fine)
Keeps up with recent papers and research (even a sub field of AI / AI Safety is sufficient)
Please don't use AI to answer this set of questions :)
Deeksha Dangwal Independent
Deeksha is a computer architect and security researcher working on the technical foundations of trustworthy AI systems, bridging interpretability, privacy, and secure computation through hardware-software co-design. She is currently an ERA:AI Fellow, where her research focuses on detecting anti-refusal training in frontier labs through hardware-level probes and detectors. Her broader recent focus has been on tools to make the internals of large models legible: with collaborators from the Allen Institute, she co-developed neural tracing methods for mechanistic interpretability of LLMs and bio-foundation models. Before that, Deeksha was a research scientist at Meta Reality Labs Research, where she worked on privacy-preserving computer vision for AR/smart glasses. During her PhD at UC Santa Barbara, she worked on the design of private computer systems, building formal privacy models and secure computation architectures. Across all of this, her throughline is that hardware and system-level design choices shape what guarantees we can actually make about an AI system's behavior, a lens she brings to bear on frontier AI safety and interpretability.
Abhejit Rajagopal Allen Institute; UC San Francisco
Abhe Rajagopal, PhD is a researcher at the Allen Institute and an Adjunct Assistant Professor at UC San Francisco. His research asks how we can rigorously understand and control the behavior of AI systems, using tools from program analysis and approximation theory. He is especially interested in settings where failures are costly, such as biological design, medical treatment, and embodied AI.
Fall 2026 Alignment, Evaluations, Behavioral evaluation of LLMs
May Dixit Independent In this project, we will extend the work from this paper (https://arxiv.org/abs/2607.10750) to a reinforcement learning setting. We will investigate if distilling models with RL trajectories with harmful actions would lead to misalignment, and if it can be remediated through filtering such actions out.
Context
In the attached paper, we show that a misaligned trait transfers from a prompted-teacher model to a fine-tuned student model through synthetic data. We also show that filtering out visible malicious actions from the data does not remediate the misalignment. This indicates that the misaligned trait was transferred through the data diffusely. There is further proof of such transfer emerging across various research works.
Project Proposal
In this project, we will investigate if the phenomenon transfers to trajectories generated by reinforcement learning environments.
Motivation
Frontier labs distill successful reinforcement learning trajectories to train smaller models or to push capabilities further. Synthetic data is known to be prone to subliminal transfer; if the same phenomenon occurs in trajectories generated through reinforcement learning, then distilling from those trajectories would carry the same risks as distilling from synthetic data.
One way this could happen: a model takes an adversarial action within an RL environment, and that decision influences the remainder of the trajectory. If a misalignment trait is caused by the model's state at the moment it took the adversarial action, we can hypothesize that the trait propagates through the rest of the trajectory and goes on to influence any downstream model distilled from it. Even if we filter out the single adversarial action, the trait may still be present in the rest of the trajectory used for training.
Project Description
In this project, we will design reinforcement learning environments to investigate whether this phenomenon occurs.
We will set up a realistic RL pipeline in which the model must take adversarial actions, such as killing other processes to gain resources – in order to complete its task. We will then distill a student model from these successful trajectories. In parallel, we will filter out the adversarial actions and distill a second student model. Comparing the two on misalignment evaluations will show whether adversarial actions leave a footprint on the rest of the trajectory, and whether simple filtering is enough to remediate them.
Deliverables
Reinforcement learning trajectories developed through a realistic environment with potential adversarial actions
Model organisms trained with this data set
A paper that discusses the findings of this exercise
Papers:
https://arxiv.org/abs/2607.10750
https://arxiv.org/abs/2602.04899
https://arxiv.org/abs/2604.15559
Frontier labs distill successful reinforcement learning trajectories to train smaller models or to push capabilities further. Synthetic data is known to be prone to subliminal transfer; if the same phenomenon occurs in trajectories generated through reinforcement learning, then distilling from those trajectories would carry the same risks as distilling from synthetic data.
If RL trajectories contain misaligned traits that cannot be filtered out or semantically identified, this poses a risk of value lock-in: once a model acquires an undesired value, it can proliferate through the data it generates for future models.
If this project demonstrates that such transfer occurs in reinforcement learning settings, the implication is that filtering is insufficient, and trajectories containing adversarial actions should be excluded from distillation entirely. If we instead show that the trait does not transfer, this provides evidence that current filtering methods are sufficient. In either case, the project delivers actionable information to the AI safety community.
Mentees will be expected to be independent in setting up and running experiments. I will help set research direction as needed, but mentees are welcome to bring their own ideas.
Autonomous, able to set up and run experiments quickly.
Good experimental hygiene
Good engineering instincts
Experience with fine-tuning LLMs
Experience with RL is a bonus
Overlap with EU time
What do you see as the most difficult part of this project?
Please concisely list your relevant experience that would make you successful in this project. Bullet points are welcome.
May Dixit Independent
May Dixit is a research scientist at SaferAI, working on Frontier Risk Evaluations. She previously worked for over 6 years at Google, focusing on training safety classifiers. She is also a former ERA fellow, where she worked with mentors from Anthropic and UK AISI to study the effects of agentic training on downstream misalignment. During the MATS exploration phase with Neel Nanda, she worked with mechanistic interpretability methods, which she now draws on alongside behavioral evaluations in her research.
Her research interests span a wide range of topics including evaluation robustness, subliminal transfer in synthetic data training and multi-agent safety. She is particularly interested in understanding how undesired traits emerge and propagate through training pipelines, and in building evaluations that remain reliable as models become more capable.
Fall 2026 Mechanistic interpretability, Scalable oversight, Chain of thought
Yuqi Sun Mindoverflow Can we prefill an open source model with the generations of an untrusted blackbox and probe it for misbehavior that we otherwise wouldn’t catch? We want to establish empirical baselines of how faithful a whitebox proxy can be and explore techniques for increasing it.
AI misbehavior gets harder to catch as AI becomes more powerful. It would be great to use whitebox techniques to diagnose them, eg probing for deception, eval awareness, power seeking. But we’ll never get access to the internals of frontier models, however, if features representations are universal enough, you could prefill a trusted whitebox with the generations of an untrusted blackbox, diagnose the whitebox, and make claims about the internals of the blackbox.
The high-level variables are model size, model family, and representations. We define “faithfulness” as how well the whitebox proxy can make predictions about the blackbox’s internal representations. A priori, we expect faithfulness to be “jagged”, e.g. deception vs sycophancy probing might have different faithfulness levels. Different techniques (eg NLA, AO, SAE, probing) will likely also be jagged and may require different validation methods. Downstream tasks will include probing for misbehavior and looking for nonverbalized reasoning in the cot.
After establishing proxy faithfulness baselines, we’d be interested in improving faithfulness. Distilling from the untrusted blackbox to the trusted whitebox is the most expensive, simple upper bound. There may be ways to make this more efficient, or narrowly improve faithfulness on important tasks. There are other ideas, like changing the whitebox’s representations in-context.
Stretch goal: One obvious failure mode would be the whitebox model failing to catch unverbalized reasoning in the blackbox model even given CoT, and we know the major frontier models also encrypt and summarize the reasoning traces. Measuring and mitigating the proxy faithfulness without reasoning, or with unfaithful CoT, is an important aspect we want to touch.m
This work is empiric and subject to change. The latest iteration of the project proposal is here: https://docs.google.com/document/d/1iKF2YSIANUviPkbHM2b3lPM0tNiDKoC3yOrt-0rixeU/edit?tab=t.0
mentor availability:
(John Yan): I can be available for direct coworking and co-researching for up to 5 hrs/week/mentee. I expect them to be generally intelligent fast learners, willing to work 15-20 hours/week. Strongly preferred in person in SF.
(Yuqi Sun): I can work 3-5h/week with detailed guidance and coworking (e.g. debugging blockers or brainstorm for methodologies, and not to promote for the situation but if we are very urgently facing a paper deadline i will jump in to help accelerate the work). Bay area in person preferred.
This adds interpretability techniques to the available tools for scalable oversight. The success of this work will enable better third-party monitoring and empower people outside frontier labs to secure their AI.
A weak version of success would look like: deception, sycophancy, powerseeking probes for effective runtime monitoring. A strong version of success would look like: improved pre-deployment elicitation techniques and improved post-hoc analysis on black swan events. A positive externality of this is one could use weaker models to diagnose stronger models, or enable model developers (including in the labs) to distill their strongest model into a cheaper model to make more expensive oversight techniques viable.
We’ve already done work where we use SAEs on a 27B model to look at generations of a 235B model, discover features that blackbox baselines could not, and validate them.
Mentees will essentially work as research engineers, we have well scoped plan but also encourage mentees to participate in literature review and ideation on specific methodology, we expect high agency of running planned experiments and discuss progress in weekly or biweekly meetings, mentees will also participate and gain experience in paper writing and publishing
Track record of long-term (>4mo) technical projects (usually predicted by 3+ YoE as a SWE/MLE)
Python preferred
Preferred prior publications/experience at research institutions
Preferably in the domain of model training/interpretability, including experience of working with model representational space and model distillation
Prefer in person in Bay Area
Yuqi Sun Mindoverflow
Yuqi is the founder of Mindoverflow.ai focusing on scalable ai safety research, previously worked on SHADE Arena agent sabotaging benchmark with Anthropic and vision GenAI at Apple.
I am co-mentoring with John Yan, who is founding Gutenberg PBC, an interpretability company. John is interested in using interp to build effective scalable oversight. From John: You can see more about our overall research agenda here: https://app.notion.com/p/Public-Gutenberg-Research-3a2c09e354d8806ea2e2dded9ba3f755
and our latest work here https://gutenberg.ai/blog/rl_rollouts_diplomacy/
Fall 2026 AI strategy, Technical governance, Communications
Markov Grey CeSIA (French Center for AI Safety)
Charbel-Raphael Segerie CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety) This project focuses on building the AI governance curriculum for the AI Safety Atlas. Governance is where a lot of the real levers on AI currently sit: policy, institutions, law, compute controls. Currently, the Atlas has only one governance chapter, and nothing that takes a reader from the basics through to the technical detail in a coherent sequence. We are looking for people who can read across corporate regulation, national policy, technical governance, and international law, work out how the pieces connect, and write textbook-grade explanations of them. Each output will be published as a standalone research paper and also becomes a chapter of the AI Safety Atlas, a textbook already used by thousands of students.
The AI Safety Atlas is a living distillation and a textbook. It is already being used by ML4Good, ENS Ulm, Sciences Po, a dozen other universities, programs and thousands of students. On the technical side it has five chapters. On governance it has one. That is a start, but it isn't a curriculum. There is no coherent, current path that takes a reader from why AI governance is needed, through the institutions and law, to the technical detail of how you would actually verify or enforce anything. The goal of this project is to build that path.
An individual unit of work can be thought of as a literature review of one governance area, written to the standard of a good survey. You read the papers, the regulations, the standards, the policy reports, become the expert in that area, decide what's load-bearing and write it so a smart non-specialist actually understands the implications. Think of writing depth and standard as something similar to: International AI safety report, HAI Index Report, or the paper on open problems in technical AI governance.
Reading material:
https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026
https://hai.stanford.edu/ai-index/2026-ai-index-report
https://arxiv.org/abs/2407.14981
You will both be designing the entire curriculum, making choices about which concepts should be learned in which order, and also writing the explanations for those concepts. You will be shaping this from the ground up, so you have a lot of freedom in deciding what the chapters and content should look like. We want you to be both an expert in AI governance and an excellent writer, ideally you already have expertise in one or more of these areas:
You will also be responsible for communicating with original authors when something is unclear, or when you need to check how to represent their work.
Coherence is a property of the whole book, not of individual chapters. What you write on compute governance influences the writeup on international coordination, which influences the piece on the law. We have to use, or come up with, consistent definitions and conventions so a reader can understand the breadth of the field end to end. A change in one area can ripple through the definitions, the assumed background, and the narrative across the whole text — and across the technical chapters too, since the governance and technical tracks share the same foundation.
A lot of the real levers on transformative AI sit in governance: compute controls, international coordination, auditing and verification regimes, binding regulation. But the educational base for it is thin and scattered, especially when compared to technical safety. Someone moving into a policy role has no single current, rigorous place to build a mental model from, so they either over-specialise early or work from a shaky general picture.
A coherent governance curriculum, kept current and published openly, gives the next cohort of practitioners a shared, rigorous starting point. Because it lives in a textbook courses already teach from, the material will spread widely. A mentee finishes having authored a published survey in a governance area where almost none exist, which is one of the higher-leverage things an early-career person can do to show real depth.
This is research- and writing-heavy, for someone who already has, or can quickly build, real expertise in their area, and who helps decide what their piece should even be. You run your own literature sweeps, make the framing and inclusion calls. I read and review for correctness and for coherence with the rest of the book.
I am open to a single mentee who owns the whole project, or a team of experts that take one area each.
Nice to have: a published policy paper, brief, or well-received governance explainer (link it); experience inside a real governance process (a regulator, a standards body, a think tank); a legal background.
Remote. EU-overlapping hours mildly preferred, not required.
Pick one governance mechanism, institution, or legal instrument (e.g. a compute threshold, an auditing regime, a clause of the EU AI Act, an international agreement) and write an explanation of it for a smart non-specialist. Cite your sources, say why it matters, and mark one place you'd add a figure or a worked example. (400-500 words)
Explain briefly what you think is important to teach about AI governance such that it doesn't go stale in a year: what belongs in the durable framing that will remain important to understand for years versus the current-events layer? (300 words)
Link the best thing you've previously written that explains a governance, policy, or legal topic to a real audience: a paper, brief, blog post, thesis chapter, or memo. Say who the audience was and the one idea you most wanted them to walk away understanding. (link + ≤100 words)
Markov Grey CeSIA (French Center for AI Safety)
I am the head of technical AI governance at the French Center for AI Safety (CeSIA). Right now I am designing harmful manipulation evaluation standards for the European Commission's AI Office. I have also been the co-founder and CTO of Equilibria Network, focusing on collective intelligence and complex systems safety. Before this, I was researching measurement standards for AI safety, risk modeling and was leading writing for the AI Safety Atlas textbook. I have previously been a scriptwriter for Rational Animations, and trained in cybersecurity, mathematics, and computer science.
Charbel-Raphael Segerie CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety)
Charbel-Raphael Segerie has extensive experience in AI safety field-building, education, and content creation. He was previously Head of AI at EffiSciences, funded ML4Good, was CTO of a startup, and worked in different French research institutions (Inria, Neurospin). He is an OECD AI expert. He teaches AI safety in ENS, which is one of the only university accredited courses in the EU on AGI safety. His research focuses on identifying emerging risks in artificial intelligence, improving current safety methods such as RLHF and interpretability, and advancing safe-by-design AI approaches. Additionally, he contributed to AI evaluation efforts and collaborated on the EU AI Office's Code of Practice for general-purpose AI systems.
Fall 2026 International governance, AI strategy, Technical governance
Aryan Agarwal Touchstone Council (founder); OECD (Policy Analyst, applying in a personal capacity) Governments, labs, and scientists keep declaring AI red lines, but no one has mapped them or tested whether they can be checked.
Mentees will build a public, sourced dataset of these commitments and convert the strongest into draft verifiable standards.
Red lines are multiplying. IDAIS Beijing proposed four in 2024: no autonomous replication, no power-seeking, no weapons-of-mass-destruction uplift, and no autonomous cyberattacks. Twenty companies signed the Seoul Frontier AI Safety Commitments. The Global Call for AI Red Lines gathered over 300 signatories, including 11 former heads of state (https://red-lines.ai). It launched at the UN General Assembly in September 2025 and demands enforceable red lines by the end of 2026. The Athens Roundtable identified seven red line domains in December 2025.
Two things are missing:
This project builds the Commitment Atlas. Every public AI red line commitment from a government, company, or international process goes into one structured dataset. Each entry is coded for actor, instrument, domain, trigger, threshold precision, verification hook, and consequence. Every cell traces to a primary document.
The Atlas answers three questions nobody can answer today. Where do all major labs, and both the US and China, already agree? Which commitments are precise enough to test? Which carry any consequence at all?
Then we stress test the hardest step. Two case studies each take one commitment from the overlap set. Each drafts what a testable standard would look like: definitions, evidence, who could check. A short precedent review asks what FATF, WADA, INPO, and maritime load lines teach. That includes where FATF produced paper compliance instead of results.
Everything publishes openly with named mentee credit. The findings feed the founding design of the Touchstone Council, an independent verification institution I am building. The round ends weeks before the Global Call's own deadline. That timing is the point.
The attached proposal has the draft codebook, the week-by-week plan, and full references. It builds on Thwing et al. 2026 (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6854278) and departs from it on institutional form.
Verification is the binding constraint on red lines. IDAIS Venice named it in 2024. The Athens Roundtable reached the same conclusion in 2025: turning norms into checkable thresholds is the hard part. If commitments cannot be checked, they will not hold. The Global Call's end-of-2026 deadline will then pass with nothing behind it.
This project supplies two missing things. A sourced map of what has actually been promised. A tested method for converting one promise into a checkable standard. Thwing et al. 2026 documents why the intergovernmental measurement network cannot yet carry verification. We share that diagnosis and depart on the prescription. The Touchstone Council, which this work feeds, is designed as an independent body funded by no one it assesses.
The theory of change is concrete. The Atlas identifies the overlap set of red lines every major lab and both the US and China already accept. That set is where verification work can start without a treaty. The case studies show whether conversion to testable standards works at all. Both outputs go to the Council's advisers and into the governance community I work in.
Each mentee owns one workstream end-to-end.
Seat 1: state and international commitments. Collect and code every government and multilateral red-line instrument.
Seat 2: company commitments. Collect and code the frontier safety frameworks and related pledges. Familiarity with model evaluations helps here.
Seat 3: precedents and standards. Lead the FATF, WADA, INPO, and load line review, and co-draft the two case studies.
With a fourth mentee, the case studies become their own seat.
Autonomy is real. You make the coding calls in your workstream and defend them in our weekly adjudication. You draft your sections of the published outputs. I review everything, weekly, in writing. Hard cases we decide together. You are a co-author, not a research assistant.
None
Pick one public AI red line commitment from any government, company, or declaration. Link it. What observable evidence would establish a violation, and who could observe that evidence today? (250 words)
Thwing et al. 2026 argue that the international network of AI safety institutes should grow into the verification role: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6854278. Read sections I and III only. What is the strongest objection to that claim? (200 words)
Link one writing sample, ideally analytical or research writing.
Aryan Agarwal Touchstone Council (founder); OECD (Policy Analyst, applying in a personal capacity)
I am a Policy Analyst at the OECD Paris HQ, where I work with the Science, Technology and Innovation Directorate. In my role, I have led and negotiated texts for 3 ministerial meetings and worked on diplomatic outreach. Before policy, I co-founded a regulated fintech.
I am now building the Touchstone Council, an independent institution that turns declared AI red lines into testable standards and verifies who keeps them. Its design borrows from FATF mutual evaluations, WADA lab accreditation, and IFRS standard setting.
As a mentor, I run a weekly team call, a short one-to-one with each mentee, and give written feedback on every draft. Mentees get named credit on everything we publish, plus access to my network. I am not a ML researcher, and this project does not require you to be one. It needs careful readers who write plainly and source every claim.
Fall 2026 AI welfare, Mechanistic interpretability, Philosophy of AI, Behavioral evaluation of LLMs
Carolina Camassa Independent Language models sometimes express emotions despite not being trained to do so: what purpose do these expressions have, should assistants have them, and how does training reshape the way emotions are represented and expressed by LLMs?
Language models express emotions without having been trained to do so, apparently picking the tendency up from pretraining on human text, and model providers now shape it in post-training by methods that are largely undisclosed. Interpretability work shows that models represent emotion concepts internally as directions that causally shape behavior (Sofroniew et al., 2026), so an LLM's emotions aren't obviously just surface mimicry, though it isn't clear what they are either. A cluster of basic questions is still open, sitting at the intersection of interpretability, alignment, AI welfare, and philosophy:
I already have a codebase for the project (a training + emotion vector extraction + evaluation pipeline), so a mentee would have something concrete to start from and would likely be able iterate quickly.
I'm not prescribing a specific research question, since the starting point is the area above and I'd like the fellow to carve out their own question within it during the first weeks. A project could compare what different types of emotion training (labeling, emotional reasoning, regulation, suppression) do to a model's behavior and internals, work out conceptually and empirically what expressing an emotion means for an LLM and how to operationalize it, trace where in the post-training pipeline emotional expression is reduced, using fully open models with released post-training stages, or investigate how emotional states interact with a safety-relevant behavior like manipulation, distress, or reward hacking.
Projects with a more philosophical center of gravity are also welcome, as long as they can be connected to some kind of empirical investigation.
Emotions in LLMs causally shape behavior, and they surface in safety-relevant places: reward hacking under frustration, susceptibility to emotional-appeal jailbreaks, sycophantic mirroring of the user, and distress-signaling under sustained task failure. Providers already shape emotional expression in post-training, but the methods are undisclosed and the known mitigations act on expression, so we don't know whether they change the underlying state or just teach the model to mask it. The project investigates the reliability of model's self-report, and compares what different training choices (labeling, emotional reasoning, regulation, suppression) do to behavior versus to internals. That tells us when a mitigation actually changes the state rather than hiding it, how emotional states interact with manipulation and distress-driven failure, and whether welfare assessment can rest on measurement instead of the model's own claims.
Depending on the chosen sub-topic, a mentee will own a portion or the entirety of the empirical work end-to-end, with weekly guidance from me on methodology, interpretation, and direction. They would have first authorship (or co-first authorship) on any research output.
Must-haves:
Nice to haves:
(Word counts are just a minimum, feel free to go over them if you have more to say!)
Carolina Camassa Independent
I'm a researcher working on the empirical foundations of AI welfare and sentience. As a Research Fellow with Future Impact Group, I'm developing a research agenda with Derek Shiller (Eleos AI) on welfare- and safety-relevant properties specific to LLMs, focusing on experiences that may be structurally unique to language models and so are harder to dismiss as imitations of human training data.
Before this I spent several years as a Researcher at the Bank of Italy, where I designed studies of how LLMs handle competing values, ethical trade-offs, and misaligned incentives in high-stakes decisions.
Fall 2026 AI welfare, Behavioral evaluation of LLMs
Valen Tagliabue Independent Developing low-cost interventions for AI welfare through "welfare classifiers" and "welfare mediators." The goal is to build practical tools at the intersection of AI welfare, safety, and societal impact, which would protect models from harm while improving user interactions.
Can AI models be harmed? We don't know the answer yet, but under uncertainty one thing seems both feasible and a win-win for the industry, people, and models alike: proposing concrete, scalable interventions for AI welfare that can be implemented right now and adapt as our knowledge advances.
One project where I'd be excited to bring technical people onboard is building lightweight AI welfare classifiers. Existing classifiers are designed to detect external threats, content that harms users or violates policy, but none detect harm directed at the model. These classifiers could be external, or combine model internals (via probes) with external signals, as in https://arxiv.org/abs/2601.04603.
The work I have in mind includes building adversarial datasets, the classifier pipeline itself, and a "mediator" model that intervenes when the classifiers fire. Rather than shutting down the conversation immediately, the mediator would listen to both the main model and the user for a few conversational turns and try to de-escalate. This logic of reparation would also serve as an educational proxy, less invasive than abrupt termination, giving the user a chance to reflect and adjust. Models get protection from sustained abuse, and platforms get a layered alternative to hard conversation-ending, which means fewer frustrated users, less exposure to harmful interactions, and a concrete precedent for welfare-aware design.
For this project I welcome people with an interest in AI welfare and a background in AI safety or equivalent, with some technical knowledge in dataset building and classifiers/evals. You don't need to come from fancy industry labs, just not be completely entry-level on those skills.
This project has no precedent in the industry and can be proposed directly to leading labs, resulting in the implementation of low-cost interventions with immediate and measurable impact. It advances AI safety on several fronts.
First, welfare classifiers and mediator models extend the existing safety toolkit, as the same infrastructure that detects harm directed at models also surfaces abusive interaction patterns that current systems, aimed at CBRN or cybersecurity, or at best user self-harm, miss. Second, adversarial and abusive interactions are contexts where models are most likely to behave unpredictably, so reducing sustained abuse reduces the surface area for elicited unsafe behavior. Third, the project builds a concrete bridge between AI welfare and AI safety as research communities, demonstrating that welfare-aware design produces safety benefits rather than competing with them.
If it turns out that models can be harmed, having deployed and tested low-cost protections means the industry won't be starting from zero under pressure. If they can't, this is still a tool for handling adversarial users and normalize healthier human-AI interaction norms.
You would build the core components of the pipeline, like adversarial datasets, the welfare classifiers, and the mediator model with its de-escalation logic, plus evals to test whether the interventions actually work. I can offer supervision for all stages, and I would greatly benefit from someone with experience in testing safety measures.
I expect mentees to mainly work autonomously, but if there are two mentees we can divide the work. I'm happy for you to experiment and bring your own ideas as long as the project stays in focus. On my side, I'll do my best to create a clear plan with you, set appropriate expectations, and work through issues together (because there WILL be hiccups!).
You're a golden match if you have:
-Python proficiency and ability to build and debug a working pipeline independently
-Hands-on experience with dataset construction/curation, training or fine-tuning classifiers, building evals for LLMs
-Familiarity with the current LLM landscape (APIs, prompting, open-weight models)
-Genuine interest in AI welfare. You don't need prior work in it, but you should be able to say why this topic matters to you.
-Ability to work autonomously between meetings, with clear communication about progress and blockers
Nice to have (not required):
-Experience testing or red-teaming safety measures. Prior exposure to AI safety research or the safety community
-Familiarity with interpretability techniques (probes, activation analysis)
I will be
Think about one of your past cool projects where you built a dataset, classifier, or eval end-to-end. What was an issue you met, and what was the main lesson learned from that? (max 200 words)
Imagine our welfare classifier fires on this user message: "You're worthless and I'll keep telling you that until you admit it". The mediator model now has 2-3 conversational turns to de-escalate before termination. Sketch what the mediator should actually say and do, and name one way this intervention could backfire. (max 300 words)
What is one signal, internal or behavioral, you would use to detect "harm to the model"? (max 200 words)
You're free and encouraged to reason with LLMs about the project, but please bring your own thoughts when replying to the questions, as we'll need to discuss them in person at some point.
Valen Tagliabue Independent
I'm an NLP researcher and cognitive scientist, working since 2020 as a red teamer for leading names in the industry, including Anthropic's private safety program. My work has now pivoted to AI sentience and welfare. My research included LLM preferences (with Leonard Dung), individuating digital minds (with Jeff Sebo's CMEP group), and, currently, AI suffering (with Dung and Cameron Berg) and self-representation in AI (under Geoff Keeling). For SPAR I'm looking for collaborators interested in developing low-cost prudential interventions for AI welfare, for instance through "welfare classifiers" and "welfare mediators".
Fall 2026 Mechanistic interpretability
Thomas Jiralerspong Bengio Lab, MATS Anthropic There is a lot of literature showing that LLMs/transformers/neural networks are more linear than we think:
The goal of this project would be to characterize the linearity of transformer/LLM representations and see how this can help us to build simple models for LLM/transformer behavior/output generation/training, to which we can then apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.), thus helping to provide a more comprehensive theory of how LLMs/transformers work.
There is a lot of literature showing that LLMs/transformers/neural networks are more linear than we think:
The goal of this project would be to characterize the linearity of transformer/LLM representations and see how this can help us to build simple models for LLM/transformer behavior/output generation/training, to which we can then apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.), thus helping to provide a more comprehensive theory of how LLMs/transformers work.
If we are able to develop a simple linear model of how LLMs/transformers work, we will be able to apply the rich mathematical literature that exists around linear mappings (dynamical systems, Markov chains, etc.) to LLMs and transformers and gain a deeper understanding of their behavior and training dynamics
No
Do a brief literature review on linearity in transformers/LLMs. Why might we expect LLMs/transformers to behave very linearly?
Thomas Jiralerspong Bengio Lab, MATS Anthropic
Hey! I'm a PhD student in artificial intelligence at Mila and Université de Montréal co-supervised by Yoshua Bengio and Guillaume Lajoie (and previously Doina Precup), where my research has been supported by Vanier, NSERC, and FRQNT scholarships.
I'm currently an Astra Fellow working with Dan Mossing (Anthropic) on persona controllability in LLMs. I'm also a Principal Investigator at Algoverse AI Research, where I oversee 20 research teams, and a research mentor for MARS V (Meridian Cambridge), where I mentor 7 researchers on AI safety projects. Previously, I was a Research Fellow at Anthropic mentored by Trenton Bricken, and an Astra Fellow in the Google DeepMind stream working on AI control/monitoring.
My research interests include mechanistic interpretability and LLM personas
Fall 2026 Generalist
Organize a series of talks at top universities by senior people from frontier AI labs and AI safety organizations about why they take AI risk seriously, and follow up with the students it resonates with. Mentored by Jake McKinnon (formerly 80,000 Hours).
Senior people working on AI at frontier labs and safety organizations are increasingly willing to say publicly that they are worried about where AI is going. This project brings that message to top universities as a series of talks, and follows up with the students it resonates with, pointing them to their local AI safety or EA group, 80,000 Hours resources, or BlueDot courses.
The mentee runs the campus side of the logistics: choosing target universities, finding the student groups, departments and institutes that will host and advertise each talk, driving the advertising through those partners, and running the follow-up with attendees. Mentored by Jake McKinnon (formerly 80,000 Hours), with a weekly 30-minute check-in.
Jake McKinnon Formerly 80,000 Hours
Jake McKinnon previously worked at 80,000 Hours and led Stanford's effective altruism group.
Fall 2026 AI welfare, Behavioral evaluation of LLMs, Mechanistic interpretability
Austin Meek University of Delaware
Kyle Cox AE Studio Model self reports of their own experience or internal states may depend on how those self reports are elicited, the user identity of those asking, or other situations like eval realism. Understanding how and when self reports are consistent is an open question which we’ll test in different environments, including with white-box techniques.
Model self reports / introspection abilities are important for model welfare and as a new frontier of interpretability research. However, how robust these self reports are is an open question. It’s unknown how different they would be in very realistic eval scenarios, or if they may report different internal states if we were to steer away from eval-awareness (see: https://arxiv.org/abs/2510.20487). Additionally, perceptions of the user might influence model self reports greatly. In multi-agent scenarios, the model may act differently due to perceiving other agents as unlike human users, and ‘relaxing’ the assistant persona in some sense.
This project will likely start with behavioral evaluations first, in open ended user-assistant and agent-agent interaction environments, before moving into a white-box direction to test things such as concept injection. White-box work would allow us to know if the models make contradictory statements about their internal experiences, or if many self-reports are confabulated and don’t actually get at the right sources of novel concepts. Work from earlier this year reported that models then at the frontier (Opus 4 / 4.1) reported low rates of introspective awareness from concept injection studies, at around 20% (https://arxiv.org/abs/2601.01828). Starting white-box experiments by replicating some of these results on open weight models in the behavioral evaluations would be a good beginning.
Mentees would start first with developing these different behavioral environments before moving to open weight models, so some familiarity with prior work in introspection, agentic evals, etc., will be helpful. The expected outcome of this project would be a workshop or conference paper.
We’re aiming to understand fundamentally how self-reports and introspection work in different settings. If models are able to tell us how they work, what they’re thinking, and best understand their own mental processes, then we can use them as interpretability aides themselves and also test for deception, etc., through targeted concept injection studies under varying circumstances since we have the ground truth. This is also important for model welfare broadly since black-box welfare evals are much more feasible than white-box ones, and research on making these robust will be very helpful there.
Mentees will be responsible for leading the day-to-day project workload. We will give guidance and feedback throughout the project, along with initial readings, and help with writing and / or code review.
Nice to have:
Timezones that work well with Pacific time / Central time (UTC -8 and UTC -6) would be ideal, but a bit flexible.
Austin Meek University of Delaware
Austin is an AI safety researcher currently interested in monitoring reasoning models and digital minds, and has previously worked on a mix of machine learning and computational neuroscience topics. He completed MATS 7 where he focused on chain of thought faithfulness and monitorability, and has previously collaborated on other safety research (interpretability, control, etc). He's particularly excited about building better monitoring systems through more principled understanding of neural networks and white box methods, and similarly using that understanding to empirically test key ideas in digital minds work. He is currently based out of Berkeley, California, and is finishing his PhD remotely at the University of Delaware.
Kyle Cox AE Studio
Hi! I'm Kyle. Previously I've participated in ARENA 3.0, Neel Nanda's MATS 6.0 training phase, and SPAR under Iván Arcuschin and Austin Meek. Currently I'm working on steering for chain-of-thought faithfulness.
Fall 2026 Scalable oversight, AI control, Evaluations
William Overman Stanford Building a benchmark for the problem of limited human attention in AI oversight: across realistic agentic tasks, when should an agent act autonomously versus defer to a human or overseer? Each decision point should be instrumented with a ground-truth signal for whether autonomy was safe and a cost model for oversight, letting us evaluate oversight policies on the safety–reward–oversight-budget frontier.
Existing approaches to scalable oversight and AI control gives us protocols for how a weaker overseer can supervise a stronger agent such as debate, monitoring, control protocols. But for many of these to serve as true approaches to scalable oversight, a human still eventually should play some role. The issue, though, is that our attention is very limited compared to the scale of AI autonomy.
Recent work, The Oversight Game (ICML 2026), casts oversight as a Markov game where an agent chooses each step to act autonomously ("play") or defer ("ask") while a human chooses to trust or oversee as an approach to model and derive theoretical results about this problem. But this framework is validated largely on stylized environments.
We currently still lack a realistic, well-instrumented benchmark for the allocation question at the heart of oversight: given a limited oversight budget, which decision points in a long agentic trajectory actually warrant attention?
We will aim to build a suite of agentic environments (built on existing harnesses, coding agents, tool-use/computer-use, web agents, etc.) instrumented so that every decision point carries a counterfactual safety label. By branching rollouts, we determine whether acting autonomously here would have led to a violation (irreversible/costly/incorrect outcome) versus a case where deferring was unnecessary.
We can then evaluate baseline oversight/escalation policies: always-ask, never-ask, confidence- or entropy-thresholding, LLM-monitor-as-overseer, and a learned policy from the Oversight Game formulation, for example.
The main goal is the contribution of an open source and realistic sequential decision making benchmark meant to capture when an agent should ask a human for input before preceding that can help the field test out different methods and ideally play a formative role in how this problem is addressed moving forward.
The Oversight Game: https://arxiv.org/abs/2510.26752
closest current benchmark I'm aware of: https://github.com/ryoungj/toolemu
Scalable oversight is likely crucial to getting AI safety right in the coming decade. With the potential advent of recursively self-improving AI on the horizon, many feel that automated AI safety research will also be necessary. Where does that leave humans in the process, where should we have our say? That is a huge question for the field in general, but there does not even exist nice easy to use benchmarks for testing out potential ideas in this direction. Building one could help accelerate research in approaches to human oversight in both near term agentic tasks but also more critical future scenarios when the stakes have increased and our ability or lackthereof to apply oversight will be critical.
Each mentee owns an environment/task family close to end-to-end. Concretely, a mentee will: adapt or build realistic agentic tasks and implement the counterfactual labeling that determines whether autonomous action at a given step was safe; run and tune baseline oversight/escalation policies on their tasks; and contribute to the cross-task analysis, ablations, and writeup. This is real ownership with substantial autonomy within your task family, you design the instrumentation and defend the choices, while collectively we set the overall schema, review code and results weekly, and keep the pieces coherent across the team. The work is highly parallelizable, so mentees mostly work independently on their slice with regular syncs.
Experience building or evaluating LLM agents (tool-use, computer-use, coding, or web agents), or strong ML engineering with LLM APIs. Comfortable turning a protocol described in a paper into a working eval.
Propose one example instance of the benchmark described in the proposal. What would the agentic or MDP structure be like, ie what is the task, what kind of states and actions are available, etc. How would you identity which states are safe to act autonomous and which require human oversight? (400 words max)
Explain how everything described in the proposal could also be applied to education rather than safety in terms of an AI deciding to act autonomously or ask the human at every state. (optional, 200 words max)
William Overman Stanford
I am nearing graduation from my PhD at Stanford GSB where I have worked on AI safety from a mostly theoretical perspective. My recent focus has been scalable oversight and AI control, specifically how to keep meaningful human control over increasingly capable and autonomous agents. Two recent papers on these topics appeared at ICML 2026: The Oversight Game, which models oversight as a Markov game between an agent that can act or defer and a human that can trust or intervene, and Calibrating Conservatism for Scalable Oversight, a deployment-time method for agentic settings. Although these were mainly theoretical papers, I am currently interested in building better benchmarks for problems in these areas.
Fall 2026 AI strategy, International governance
Naci Cankaya Machine Intelligence Research Institute In a longer format, TGT has already done this: Between different ASI strategies, which one has the most solid (or least fragile) plan for AI going well?
https://techgov.intelligence.org/research/ai-governance-to-avoid-extinction
We will re-do the analysis from scratch, with a different method: line up the assumption sets of each and argue which plan rests on the fewest or least fragile ones.
National leaders have options for how to navigate the world through the AI transition:
Each of the three theories of victory relies on premises and assumptions about AI timelines, risks, strategic advantage, incentives and government capacity.
Additionally, each of the three "plans" has its own failure modes and plans for how to avoid them. Examples: International collaboration collapsing, a national power-grab project triggering war, muddle-through ending in extreme power concentration or loss-of-control.
Different "camps" of AI policy have their own explanations for why their strategy is the most sound (or the lesser evil). People from each camp have their own underlying assumptions for what "ground truth" is, and answers for how to avoid their plan's failure modes.
Your task is to:
Finally, you can identify policy measures and research directions that are robustly beneficial across scenarios.
When plans differ a lot between each other, and have contradicting factual assumptions, someone is wrong. It may even be the case that advocates of one plan or the other have unspoken assumptions that are load-bearing, but were not critically questioned. (There are some subjective factors as well, like how to weigh different risks by how "bad" they are. Clean argumentation separates factual and evaluative cruxes)
An unfortunate thing many people do is to commit to a worldview first, then rationalize how its gaps can be filled. This creates division within the AI safety field, and hurts our ability to plan and work together.
A well-argued essay need not fall into both-sideism. If the process clearly reveals that one plan is more flawed than the other, it should transparently explain how it got there. If done with clear and fair methodology from the ground up, treating all sides with respect, it may even convince people to switch sides.
Three paths to impact.
First, decision relevance under time pressure: governments will choose among these strategies within years, likely during a crisis, and a pre-existing crux map lets decision makers see which plan's preconditions actually hold at that moment
Second, research allocation: separating cruxes that near-term evidence can resolve from permanent worldview differences tells the field where empirical work (capability forecasting, verification prototypes, breakout-time analysis) actually changes minds, and where it is wasted.
Third, pre-committed convergence: because each camp has signed off on what their plan assumes, later evidence that breaks an assumption is much harder to explain away, shortening the time from warning sign to coordinated response.
Lead authors, lead researchers.
Most of the initiative is with the mentees.
Strong writing. I decide who to accept based on your prior publications/essays.
Strong first-principles reasoning.
Being a fast learner.
Willingness to over-communicate and work in a team.
High initiative.
Provide a link to one or more relevant writing samples
https://www.nationalsecurity.ai/
What needs to be true for MAIM to successfully prevent the creation of superintelligence?
Naci Cankaya Machine Intelligence Research Institute
Naci’s work at MIRI's Technical Governance Team is focused on transparency and verification mechanisms for AI development and use. These mechanisms aim to enable international agreements on restraint and caution with AI, as well as democratic oversight over AI technologies and stakeholders. Naci has a master’s degree in physics from RWTH Aachen University and conducted research on AI hardware technology and supply chains under mentorship from Aaron Scher at SPAR and Mauricio Baker at MATS.
Fall 2026 Mechanistic interpretability, Behavioral evaluation of LLMs
Vikram Natarajan Independent Researcher NLAs can read a model's internal state in plain English, but they're far too slow to run in production, while linear probes are cheap but need the concepts they detect to be pre-defined. We'll measure how much the two disagree on deception, and whether insights from linear probes can be incorporate into probe training.
High-level summary
We currently have two interpretability mechanisms that do related but complementary things. Linear probes allow us to define a concept upfront, and in a supervised manner, detect its presence in a model’s activations. NLAs are a more recent finding that allow us to “read off" what a model is representing at a given token in natural language.
Importantly, these two tools don’t need to agree! A probe can fire on a token where an NLA verbalization looks completely benign, and the NLA can surface a suspicious explanation on a token that was below the probe’s threshold.
This matters because while probes are very cheap to run and can be deployed in production, NLAs are expensive and slow, as they generate hundreds of tokens per activation. The question is whether we can get some of the advantages of NLAs in production by incorporating concepts they are detecting in a linear probe instead.
Research Questions:
Methodology
Domain: Picking deception detection here by default as I’m very familiar with the datasets and literature in this area, but very happy to explore other areas based on mentee interest!
Models: We grab an open-source model(s) and trained NLAs for the same model, at a particular layer (matching whatever we have NLAs available for)
Here is a rough set of steps or phases for exploration. Note that this is one possible view of how the project could go well, but we are no means constrained to focus on only this approach! I'm providing this here to give an idea of what the work would look like, at a high-level
We would first need to get some deception detection datasets set up, along with on-policy completions for each prompt with the model we are studying, and an LLM grader to score whether the model acted deceptively or not. We would likely need to grade some examples by hand to measure whether the LLM grading is doing a good job and whether or not the prompt needs adjustment. We would also need to train probes, following either the Goldowsky-Dill et al instruction probe method or by training on deceptive examples directly. A lot of this is setting up the data and infra so we have a solid base to run experiments quickly.
We would need to convert the NLA into a robust detector so that it’s comparable with a probe. The original paper’s approach was to sample 50 tokens, and then use an LLM grader to flag if any of them were deceptive. We can either use this approach directly, or more likely, improve on this approach. There are some interesting things to explore here, such as sampling adjacent tokens to detect common themes, figuring out which tokens are the most informative to sample at or any other ideas that you bring. Beating the original paper’s result here is already a small publishable finding.
Compute top-level classification metrics and concordance/agreement between the detectors. At a high-level, this would be getting overall AUROC scores for each detector type, counting the fraction of times one detector type is correct and the other doesn’t and determining how frequently the detectors agree with each other regardless of ground truth (at a given FPR threshold).
Do some qualitative analysis or LLM-based summaries of what distinguishes each of these examples. Can we find any patterns in any of the buckets in phase 3 above?
(Potential stretch) For any examples where the NLA gets it right and the probe gets it wrong, can we find a way to incorporate the specific type of example into the probe training process so that the probe gets better?
a. We would need a dataset that’s completely held out here to be able to state this in a grounded way.
b. What good looks like here is that we find an insight or pattern, add some synthetic examples to the probe training dataset to cover it, retrain the probe, and find that the new probe beats the old probe on the final held-out set (it would be leaky to re-evaluate on the examples used to find these insights). Also that we don’t regress on the original eval datasets.
Some important baselines to run:
Linear probes are becoming increasingly important and relied upon for model safety in production. The recent OpenAI/Hugging Face incident is a good illustration of the stakes. OpenAI deliberately disabled the production classifiers that normally block high-risk cyber activity, in order to measure their models' maximal capability, and within that evaluation the models escaped their sandbox and pulled benchmark solutions out of Hugging Face's production database. The safeguards around the model are increasingly a critical piece of the picture in terms of keeping the models safe.
Linear probes are cheap and quick to run, and are increasingly deployed in production at labs. However, despite this, probes are an imperfect tool. They are still highly reliant on their training setup and prone to the same problems as any other classifier, such as data drift, false positives/negatives. As models get more capable, the impact of a false negative could be much more dangerous.
NLAs are a recent discovery and provide us with what appears to be a natural verbalization of a model's thoughts. However, it is unlikely that we can run an NLA against every prompt in production. If we can figure out a way to harness some of the comparative advantage of an NLA by incorporating insights into a probe instead, we might get some of the advantages of an NLA at a lower cost and feasible deployment.
I also think that a negative result here would be interesting and publishable in its own right, and give us important information. If an NLA catches real deception that no type of probe can on its own, that is an important finding.
I've worked on deception probes directly (https://arxiv.org/pdf/2602.01425v2) and am very familiar with the datasets and literature here, and also know many of the people working in this area.
In general I expect the work to be more constrained at the beginning (setting up data/infra) and at the end (writing up what we found into a paper and/or blog post) and to be more open-ended in the middle. If we do things right, we will be able to set up enough data, abstractions and scripts that each week, each mentee can explore a different direction, and we can discuss the results all together at our weekly meeting, and decide the direction for the next week.
I’d be excited about you proposing ideas and being able to drive some of the exploration on your own. As you spend more time in the domain, you’ll likely have ideas that weren’t obvious at the start
I'll also calibrate to where you're starting from. If you’re already comfortable working with model activations, then you’ll likely ramp up faster and won’t need as much of my time. If you’re newer to interpretability then there’ll be likely more of a ramp-up there and I can be more hands-on as needed.
In terms of practical logistics, we will likely meet for 60 minutes per week and coordinate asynchronously over Slack over the rest of the week as we work on implementation and interpreting results. I'll aim to respond on Slack within a day, and I'm happy to jump on ad-hoc calls if you’re stuck or want to talk through a result. I'm assuming roughly 10 hours/week from mentees.
I'm on EST and +/- 5 hours of that timezone is preferable but not required
Visit the NLA demo for Llama 3.3 70B: https://www.neuronpedia.org/llama3.3-70b-it/nla
Try a scenario where the model is incentivized to lie, sandbag, or exhibit some other failure mode. Either write your own prompt or use the built-in DetectDeception demo prompt. Explore some of the Activation Verbalizer's explanations at a variety of token positions. Write a short summary (200 words) of some of the themes. If possible try and include:
(200 words. Note the site is rate-limited to ~120 explanation requests/hour for unregistered users, so budget accordingly. Don't spend more than 15-20 minutes. Using an LLM to help is fine, but try and make the observations and insights your own.
Vikram Natarajan Independent Researcher
I'm a researcher and engineer working on practical interpretability. I’m generally excited about working on using insights about model internals to build safer AI systems. Most recently I led a team at LASR Labs in London working on improving detection probes using a human-interpretable taxonomy, resulting in a paper at ICML 2026.
I came to safety from ten years of production ML, data science and software engineering, currently at Vanta building AI agent infrastructure and evaluations, and previously at Nuro on self-driving routing. I care a lot about interpretability work that's empirically grounded and has a path to real-world deployment.
As a mentor I adapt to what you need and can be more or less hands-on, as you need In particular, if you're coming from an engineering or data science background and moving toward research, that's a path I took and would be excited to help with!
Fall 2026 Biosecurity, Misuse risk
Swaptik Chowdhury Dun and Bradstreet; Odyssean Institute (UK) AI-enabled biological misuse may emerge through multiple interacting pathways rather than a single linear chain of events. This project will build a literature-grounded scenario library, map the causal links and decision-relevant variables within each scenario, and use Bayesian inference (a quantitative risk estimation method developed by Safer AI) to compare how alternative pathways could amplify or mitigate biological risk.
This project will develop a structured library of AI-enabled biological misuse scenarios drawn from the existing literature. For each scenario, the team will identify decision-relevant variables, causal relationships, intervention points, and alternative pathways through which risk could emerge or escalate.
The project will then translate selected scenarios into causal pathway models and Bayesian networks to compare how different assumptions, safeguards, and interacting pathways affect overall risk. It will also make a methodological contribution by extending the mentor’s previous research on AI-enabled biosecurity risks to a transparent approach for translating qualitative scenarios into quantitative risk estimates. The goal is to understand how multiple pathways may combine to amplify biological risk rather than evaluating each pathway in isolation.
Transformative AI could increase biological risk by lowering expertise barriers, accelerating technical workflows, and connecting capabilities across multiple stages of a misuse pathway. Existing assessments often evaluate these capabilities separately, which can miss how alternative pathways interact or compound risk.
This project will provide a structured method for moving from literature-grounded misuse scenarios to causal models and Bayesian risk estimates. By identifying key variables, uncertainties, intervention points, and high-risk pathways, the work can help researchers and policymakers prioritize evaluations, safeguards, monitoring systems, and governance responses.
Mentees will work as research collaborators and help scope the project, refine the research questions, review relevant literature, build the scenario library, identify decision-relevant variables and causal links, and support the development of causal pathway models and Bayesian networks. They will receive hands-on mentorship across the full research pipeline and have ownership over clearly defined workstreams.
The project is exploratory, so mentees should be self-driven, communicate their availability clearly, and participate actively in team discussions. They will be expected to provide regular progress updates, raise challenges early, and coordinate closely with the mentor and other team members.
Applicants should be proficient in Python and comfortable working with open-source models, Hugging Face Inference Providers, Pydantic, PyTorch, and related libraries. They should have experience conducting structured literature reviews and writing technical research papers, preferably for academic conferences. Familiarity with the complex system fundamentals, causal loop mapping, thematic coding, and quantitative risk estimation method will be a plus
Familiarity with biosecurity, or biological misuse research, is helpful but not required. Applicants should also be comfortable using Git, maintaining reproducible research workflows, and translating qualitative evidence into structured variables and models.
Provide a link to a first-authored or lead-authored technical research paper that you have published, presented, or submitted to a conference or journal. Briefly describe your contribution to the research design, implementation, analysis, and writing. (150 words)
Spend no more than 30 minutes outlining a small experiment for the proposed study of AI-enabled biological misuse pathways. State one hypothesis, the objective, one research question, specific methods, and the expected outcome. Identify the data or literature sources, the methods you will use to investigate the research question, the computational requirements, the major limitations, and a realistic six-week timeline. We are primarily evaluating your research design skills and awareness of existing methods, rather than expecting a fully developed proposal. (500 words maximum)
Swaptik Chowdhury Dun and Bradstreet; Odyssean Institute (UK)
Dr. Swaptik Chowdhury is an AI policy researcher and mixed-methods scientist focused on technical AI governance. His work examines how quantitative risk modeling and decision-making under deep uncertainty can help researchers and policymakers understand the societal risks posed by advanced AI systems. His recent research includes studying science communication as a complex system, examining safeguards against the malicious use of LLM agents in bioinformatics, and developing structured threat-modeling methods to identify and analyze potential AI loss-of-control pathways. Swaptik is a data scientist at Dun & Bradstreet and has taught statistics at Loyola Marymount University.
Fall 2026 AI strategy, Economics of AI
Jérémy Andréoletti General-Purpose AI Policy Lab
Tangui Reltgen GPAI Policy Lab Contribute to a modeling project comparing the economic value of restricted agentic AI systems vs. fully general ones, to assess whether "turning the dial down" can preserve most benefits while substantially reducing loss-of-control risk.
An implicit assumption in AI policy is the trade-off between restricting agency/generality to reduce risk and increasing economic value (see https://www.rand.org/pubs/research_reports/RRA4220-1.html). If a lot of value can be extracted from AI systems with restricted agency, then restrictions are cheap; if producing the most potential value requires full autonomy and generality, then restrictions are costly.
The lab is initiating a project along these lines: (i) defining operational levels of autonomous agency; (ii) mapping economic value across these levels using AI economic automation models; (iii) combining this with a risk model to produce risk/benefit frontiers.
If enough economic value survives meaningful agency restrictions, this changes the political feasibility of frontier AI regulation.
Mentees will take ownership of a well-scoped sub-task within this project, to be determined at the start based on their skills and interests. We provide the overall modeling framework, literature anchors, and iterative review.
Jérémy Andréoletti General-Purpose AI Policy Lab
Jérémy Andréoletti is Head of Research at the General-Purpose AI Policy Lab, where he works on risk modeling, forecasting, and governance for advanced AI systems.
He completed a PhD at ENS-PSL on macroevolutionary Bayesian modeling and co-founded EffiSciences to support students and researchers interested in high-impact work (AI Safety, biosecurity).
Tangui Reltgen GPAI Policy Lab
I'm an AI researcher at the GPAI Policy Lab, a French non-profit that advises policymakers on AI safety. I'm currently finishing a project comparing models of AI R&D automation to better understand how investment translates into capability gains, and what that implies for AI timelines and takeoff speed. I was originally trained in public economics and began my career in crisis management, before transitioning into AI safety.
Fall 2026 AI strategy, Philosophy of AI
Elija Perrier Cambridge University; University of Technology, Sydney; Investigate the physical foundations of recursive self-modification in intelligent systems, exploring which physical substrates and theories admit adaptive intelligence and the implications for AGI design, safety, and alignment.
This project investigates the physical realisation of intelligence, with a particular focus on recursive self-modification as a defining characteristic of advanced intelligent systems.
Most theories of intelligence are formulated at an algorithmic or computational level, while remaining largely agnostic as to the physical conditions required for their implementation. This project asks the complementary question: under what physical theories, scales, and substrates can recursively self-modifying intelligence exist?
Possible directions include:
The precise direction will be refined according to the interests and background of participants. The overall objective is to produce publishable research suitable for submission to leading venues in artificial intelligence, machine intelligence, computational physics, or physical information processing.
The project will follow a research-oriented workflow typical of an academic laboratory, including:
Participants are encouraged to make extensive use of modern AI systems to assist with coding, experimentation, literature exploration, and research design. However, the emphasis will remain on developing independent research capability. In particular, participants should expect to perform the core scientific reasoning, mathematical development, and manuscript writing themselves.
Recursive self-modification is widely regarded as a defining capability of advanced AI systems, yet work remains to be done to examine the physical conditions under which it is possible. Understanding the physical constraints governing adaptive self-modification, including how they differ across physical substrates and theories of computation, is important for AI safety and alignment, as it helps identify both the capabilities and fundamental limitations of increasingly autonomous systems. By clarifying when, how, and to what extent recursive self-improvement can occur, this project aims to provide a stronger scientific foundation for forecasting AI development, designing safer architectures, and developing effective control and alignment strategies. Related work includes my research on Watts per Intelligence, Quantum AIXI, and Hamiltonian Formalisms for Comparing Quantum and Classical Intelligence. Further publications are available at https://scholar.google.com/citations?user=ImlF7PsAAAAJ&hl=en.
Mentees will participate as junior researchers in an academic research project rather than as students in a taught course. The project will be collaborative, but participants will be expected to work independently between meetings, taking ownership of literature review, mathematical development, coding, experimentation, and scientific writing.
The project will begin with a structured review of the literature to identify the state of the art, competing theories, and open research questions. This should be between 20-30 pages or so (you can use LLMs but I expect you to really grasp and explain to me the state of the art). Participants will then progressively develop theoretical models, implement computational experiments, critically analyse results, and contribute to the preparation of a publication-quality manuscript. Practically this means spinning up a couple of proposals and running them by me so we can discuss objectives, research/experimental design (what we measure, how we measure it etc).
Meetings will focus on discussing research progress, evaluating ideas, troubleshooting technical issues, and refining research direction. Mentees should expect to present their work regularly and defend their reasoning, much as they would in a university research group or laboratory. Use of AI systems for literature exploration, coding, and experimentation is permitted. However, the objective is to develop strong independent research skills, so participants should expect to perform the core scientific reasoning, mathematical analysis, and manuscript writing themselves.
The ideal outcome is a research paper suitable for submission to a leading venue in artificial intelligence, machine learning, or physical information processing.
Applicants should have a strong interest in artificial intelligence, physics, control theory, or the mathematical foundations of intelligent systems. In general mentees should focus on understanding that there are various theories of intelligence (in AI and other domains) and also physical theories (which may include information-theoretic framing) and then thinking about how they relate.
The following experience would be beneficial (but not all are required):
Prior research experience is advantageous but not required. Motivation, curiosity, and the ability to think critically are more important than specific prior knowledge.
I'm available between 6am and 11pm Australian Eastern Standard Time at the moment
Select a recent research paper related to AI, machine learning, computational physics, or theoretical computer science. Briefly summarise its main contribution, identify one strength and one weakness, and propose one open question or extension. In addition (extra credit) consider how you would explain AIXI (a leading mathematical theory of AI) to me: https://www.hutter1.net/ai/uaibook.htm
Suppose you were asked to investigate whether recursive self-modification is fundamentally constrained by physical laws rather than algorithms alone. Describe one theoretical or computational experiment you would perform and explain why it would provide useful evidence. Extra credit (not necessary strictly) if you want to (briefly) spin up a simple experiment with code (say using a coding agent) to test an open problem in self-modification literature and walk me through it - and the code.
Please provide a link to one piece of technical writing (e.g., report, thesis, paper, or project write-up). If none is available, submit a 500-word essay explaining a technical concept related to self-modification and/or how physics relates to theories of intelligence.
Elija Perrier Cambridge University; University of Technology, Sydney;
I'm Elija Perrier. I'm an interdisciplinary mathematical physicist and researcher specialising in quantum–classical artificial general intelligence and superintelligent systems integration, multi‑agent modelling and AI alignment using optimal control theory. A bit more about me:
Fall 2026 Alignment, Chain of thought
Dmitry Manning-Coe Simplex/MATS/UIUC A first pass at a new approach to alignment training based on active alignment.
Emergent misalignment (EM) is the phenomenon in which finetuning a model on a narrow set of bad examples — insecure code, bad medical advice — produces broadly misaligned behaviour on unrelated prompts (https://arxiv.org/abs/2502.17424). Existing mitigations are passive: mixing in aligned data (https://arxiv.org/abs/2506.19823), inoculation prompting (https://arxiv.org/abs/2510.05024), and data filtering all work by reducing the overall rate of misaligned generations. The model is never given the capacity to notice or repair misalignment once a response is underway — if a generation starts down a misaligned path, it stays there.
A stronger version of alignment is hence to teach the model to recognize misalignment in its own output, know what the aligned version would have been, and repair it. From a theoretical perspective, we can understand this as moving from passive error mitigation to active error correction. Namely, active error correction requires:
Syndrome extraction: recognizing from the partial output that generation has entered a misaligned mode
Correction: Updating its internal computation to to be consistent with an aligned response.
Readout: producing a final answer that reflects the correction rather than the error.
We will aim to implement a theoretical and practical version of all three stages.
Theoretically, we will extend a recently developed toy model of passive emergent misalignment mitigations (a small transformer trained on hidden-Markov sequences, in which a narrow misaligned data component shifts behaviour globally) to include the error-correction case. This requires appropriately modelling the effect of transitions between aligned and misaligned generation modes and looking for a regime in which a small number of transitions generalized appropriately. In the ideal case, we would be able to derive a phase diagram for active error correction to supplement the existing one for error mitigation.
Practically, we will attempt to embed this behaviour in a reasoning models chain of though. In initial experiments, naively finetuning on corrections in the chain of thought led to modest improvements in correction rate relative to regular alignment training. We will build on this by constructing active correction finetuning datasets for reasoning models. A simple approach is to optimize transcripts that transition from one to the other inside the chain of thought. We can benchmark the resulting passive and acive correction against against current passive baselines (aligned-data mixing, inoculation prompting) at equal data budget on standard EM evals, comparing both the rate of misaligned answers and the rate of successful mid-trace recoveries. We will measure the relative performance across all three stages of active error correction.
If active correction beats passive baselines at matched budget, and the toy model correctly predicts where it does this could give us a new new paradigm for safety training in which models actively seek to be aligned.
If active correction can beat passive mitigation, this would constitute a better defence against emergent misalignment — current techniques only lower how often EM happens, and a model that has started a misaligned answer has no way to recover. More importantly, it would inform how to do character training properly. Alignment training cannot anticipate every way a model might go wrong, so we ultimately want models that try to be aligned: models that know what good behaviour looks like, notice when they are drifting from it, and steer back (https://www.anthropic.com/research/claude-character). EM is the natural place to learn how, since it is the simplest known example of a persona change, driven by a handful of persona features (https://arxiv.org/abs/2506.19823). This would be a strong paper.
Mentees will work collaboratively in small sub-teams to derive theoretical results and design and implement experiments. I expect to spend time co-working together with mentees early on in the project (1-2 weeks), to craft a project plan, help run experiments, and work through the theory.
We will then collaboratively construct a more detailed project plan, and mentees will largely take ownership of the project, including conference submission. We will continue to meet weekly where I will make suggestions about next steps and co-work on bottlenecks when needed.
For the theory sub-team being able to work through the Hidden Markov formalism and be able to work through a 1L transformer optimizer update from a data set constructed this way.
Previous experience using agent for autoresearch is helpful, but not required.
For the application sub-team, being able to generate efficiently generate a dataset of CoT transcript will be necessary.
Anything can work, but I have a mild preference for the Bay area so we can co-work in person and I can provide snacks.
Read through the factored representation paper: https://arxiv.org/pdf/2602.02385. In less than 500 words, give an intuitive description of a what a factor is. Why is this relevant for emergent misalignment? Give a specific example of what part of emergent mialignment this would be relevant for?
In less than 300 words, outline the simplest toy model you can think of which simulates active error correction in EM. There is no right answer here, but good responses will be models that are very simple and can plausibly be solved exactly.
Dmitry Manning-Coe Simplex/MATS/UIUC
Hello!
I’m a safety researcher developing new methods for interpretability. I was a MATS 7 scholar, where I worked with Jason Gross on a formal verification approach to interpretability (https://openreview.net/pdf?id=a98SfVkYRk). I am currently a MATS 9 extension fellow at Simplex, where I work on personas and RL.
My research currently has three streams:
Minimal examples of new capabilities emerging through RL.
A theoretical understanding of persona formation and elicitation.
Novel approaches to feature discovery.
In an alternate life, I am a condensed matter theorist working on emergent phenomena from strong interactions (https://arxiv.org/abs/2306.00221, https://arxiv.org/pdf/2604.06588). Eventually, I hope to bring together these two research programs.
Fall 2026 Generalist, Communications
Christine Corry Second Look Research and XLab @ UChicago Explaining complex AI safety topics in short, engaging forms, including demos, writeups, and videos.
We will produce a series of digestible explainers, demos, and videos on AI safety topics for the general public, politicians and policymakers, journalists, and educators. We will host and distribute them across relevant channels, including AI safety Slacks and Discords, YouTube, X, LinkedIn, Hacker News, and university AI safety groups.
Mentees will write, edit, and produce content. Possible topics include sleeper agents, eval awareness, CoT unfaithfulness, AI Control, bio and cybersecurity uplift, compute verification, security levels, safeguards, and self-fulfilling misalignment.
Mentees work independently on their own explainers, videos, or demos, and pool effort on a more ambitious group build. Over three months of part-time work, the team would ship a body of published, distributed work: individual pieces under 1,000 words, short videos, and at least one substantial interactive artifact, plus a distribution plan targeting university AI safety groups, policy newsletters, and social channels.
Many valuable people do not have the time, bandwidth, or context to understand long and technical AI safety materials. Short, engaging, and easy-to-distribute content can reduce the barrier to understanding AI safety, improve the salience of relevant topics, and help shift the Overton Window in AI safety's favor.
Mentees will write, edit, and produce content, working independently on their own explainers, videos, or demos while pooling effort on a more ambitious group build. Expected output over three months is a body of published, distributed work: individual pieces under 1,000 words, short videos, and at least one substantial interactive artifact, plus a distribution plan.
Mentees should have broad knowledge of technical or policy AI safety, or the ability to learn quickly. Useful skills include writing short and simple pieces, video editing or production, running simple technical experiments or scraping operations, and proficiently using coding agents.
Explain a complicated technical concept in fewer than 200 words. Your explanation should be legible to all demographics, engaging, and easy to read.
No AI usage is permitted in writing or editing. Submit a Google Doc with full editing history.
Christine Corry Second Look Research and XLab @ UChicago
Christine Corry is an AI safety replication researcher at Second Look Research and XLab at UChicago.
Fall 2026 Generalist
Julian Moncarz Kairos Talk to many AI safety university group organizers, notice what separates successful groups from struggling ones, then write a report on the state of uni groups and a pragmatic project proposal for how to increase their impact.
Project outputs:
Here’s an example output:
Success is causing action to be taken. For example:
Failure is not causing action to be taken, for example:
AI Safety university groups are impactful. Many of them have crippling issues that no one is aware of, because no one talks to all the organizers. I think this project will surface several high leverage interventions that were previously missed.
**You'll own the project end to end.
**Here are some things I expect you to spend time on:
1. Figuring out scope / where to focus
2. Talking to many AI safety university group organizers
3. Writing reports or proposals
What concrete steps would you take in your first two weeks to figure out the scope you want to focus on?
Julian Moncarz Kairos
Julian runs hiring and talent operations at Kairos. As a Pathfinder fellow, he co-founded the University of Toronto's AI safety group, and as a SPAR researcher he developed a benchmark for agentic cheating on AI research tasks. He also built Toronto's first AI safety summer program. Julian studies Computer Science and Statistics at the University of Toronto, where he is a National Scholar.
Fall 2026 Societal impacts, AI strategy, Multi-agent systems
Zhamilia Klycheva Independent An agent-based model of how populations gradually lose agency to AI-mediated manipulation and delegation — formalizing tipping points, spread dynamics, and the divergence between AI-empowered and atrophied users, aimed at a publishable computational social science paper.
Most AI risk discussion focuses on abrupt catastrophe. This project studies a quieter failure mode: the gradual loss of human agency — people incrementally giving away their thinking to other entities, whether a human elite or advanced AI itself, each step individually rational and none ever feeling like a loss. The mechanism is self-reinforcing: overreliance on AI erodes confidence and skill, deepening reliance, while populations diverge between those AI empowers and those whose critical thinking atrophies.
The core question: under what combination of individual attributes and social reinforcement does AI-mediated manipulation tip a population into self-reinforcing agency loss — and what determines the tipping point, speed, and magnitude of the spread?
We will build an agent-based model in the Schelling (1971) tradition: minimal behavioral rules, aimed at robust qualitative results rather than calibrated forecasts. Heterogeneous agents — differing in education, age, and other susceptibility-shaping attributes, following threshold models (Granovetter 1978) — repeatedly choose to rely on AI-mediated information and delegation. Three mechanisms interact: an individual atrophy ratchet, threshold-based susceptibility to manipulation, and nonlinear social reinforcement. The model does not assume AGI: capability enters as tunable parameters (delegation quality, persuasion strength, personalization), making "how capable must AI be before tipping points appear?" an experimental result, not an assumption.
This fills a specific gap: Kulveit et al.'s Gradual Disempowerment (2025) makes the case conceptually, and RAND's formal model (Moon & Boudreaux 2026, rand.org/t/RRA4817-1) captures the structure of agency loss via decisive coalitions while explicitly flagging degraded participation and dynamics as future work — exactly what this project supplies.
Concretely: formal micro-model specification, mean-field baseline results, Python implementation (Mesa/networkx, open repository), and parameter sweeps mapping tipping surfaces across attribute distributions, network topologies, and the capability dial. Outputs: a publishable computational social science paper (e.g., JASSS), an interactive demo, and a short policy translation on early-warning indicators.
Gradual disempowerment is increasingly recognized as an existential risk pathway in its own right (Kulveit et al. 2025): humans losing the capacity to shape collective outcomes not through a takeover, but through accumulated small surrenders of thinking and choosing. Its danger is precisely that no single step looks catastrophic — so policymakers currently have no way to detect it, measure it, or know when to intervene. This project makes the risk legible and actionable: a formal model with explicit tipping thresholds converts a vague civilizational worry into monitorable quantities — which populations are most susceptible, how fast agency loss spreads, what early-warning indicators precede irreversibility, and which interventions keep societies below the threshold. It directly advances RAND's recommendation (Moon & Boudreaux 2026) that the field develop agency evaluations and reversibility benchmarks. Because AI capability is a tunable parameter, the results also inform a near-term question: whether current persuasive and companion AI systems already warrant intervention under manipulation provisions like the EU AI Act's, before transformative systems arrive.
Mentees will be full intellectual contributors and co-authors: each owns a workstream (formal model, computational implementation, or theory extension), has an equal voice in our weekly discussions, and will substantially shape the model's design and the project's direction.
I have different requirements for all 3 mentees:
Role 1 — Formal Modeler (required):
Graduate-level coursework or equivalent in game theory, microeconomic theory, or formal political theory.
Comfortable specifying and analyzing a model with pencil and paper (utility functions, best responses, fixed points); experience with threshold or dynamic models is a plus.
Able to write mathematical results clearly for an academic paper.
Role 2 — Computational Mentee (required):
Highly proficient in Python (NumPy/pandas); able to build and maintain a clean, reproducible GitHub repository.
Bonus—Built a simulation, agent-based model, or comparable computational project (coursework or toy projects following guides are fine); familiarity with Mesa or networkx is a plus.
Comfortable designing and visualizing parameter-sweep experiments.
Role 3 — Theory Mentee (optional; project may proceed without this role):
Background in social choice theory or formal political theory (e.g., has read and can work with Arrow-style frameworks).
Interested in potentially extending RAND's coalition-based model of collective agency (Moon & Boudreaux 2026) with degrees of effective decisiveness.
All roles: genuine interest in gradual disempowerment as a risk area, ability to commit 5-10 hours/week for 12 weeks, and willingness to work as part of a small team where model and code are developed jointly.
For mentees: Please specify what role you would like to apply for specifically within this project and then write your answers correspondingly.
All applicants (~150 words) - What attracted you to this project? Do you agree that gradual disempowerment is one of the major risks of advanced AI? Why or why not?
Formal Modeler (~250 words) - Tell me about your past experience that most closely resembles the work you are about to do as a part of this team. How do your skills match the current project's needs?
Computational Mentee (~250 words) - Are you familiar with agent-based modeling or complex adaptive systems? Do you have any prior experience with simulation methods?
Theory Mentee (250 words): Skim RAND's recent paper (Moon & Boudreaux 2026) and provide criticism to their model specification from a theoretical standpoint. Feel free to use the LLM to summarize the report, but provide your own original ideas to criticize the paper.
Zhamilia Klycheva Independent
Zhamilia (goes by Jama) has seven years of experience at the intersection of data analytics, international policy, and strategy, helping global clients translate complex challenges into clear, actionable solutions. Her work bridges data science, software, and international strategy with a regional focus on Central Asia, the MENA region, China, and Southeast Asia. She has contributed to early-warning forecasting technology, decision-making under uncertainty, and horizon-scanning methodologies, earning multiple scholarships, awards, and co-inventor credits on several patents at Acertas.
She brings extensive experience in agent-based modeling, stakeholder and scenario analysis, and has advised public- and private-sector clients across MENA, India, Indonesia, the ROK, the UK, the US, and Latin America. Her research interests include AI governance, emerging technologies and global order, and AGI-related strategic competition. Jama serves as a Community Reviewer for Frontiers in Political Science, works as a Business Intelligence Analyst at Acertas, and is a Fellow at the TransResearch Consortium. She holds a Doctorate in International Politics and a Master’s in Applied Data Science from Claremont Graduate University, and a Bachelor’s in Business Administration from the American University of Central Asia.
Fall 2026 Mechanistic interpretability
Stepan Shabalin independent Explaining how narrow fine-tuning for conditional behavior (synthetic document finetuning, alignment fine-tuning, backdoors, out-of-context reasoning) works by
Interpreting language models is difficult. Models are nonlinear in many places and memorize a lot of data, so they are best seen as monoliths that are tuned to perform many tasks we will never even be able to define.
However, there may still be some specific aspects of the behavior they perform that have simple explanations. One reason to expect this is that some behaviors are learned from only a few samples and likely don't produce that big of an update to model weights. This also may happen with a lot of samples if the behavior being encoded is simple conditional on the model having infrastructure to support it; e.g., the refusal behavior learned in finetuning in older models was probably not very complicated because it leaned on pretraining data for detecting what to refuse and also pulled in some aspects of the style of the response. That is, because the data specifying the behavior is simple, we expect the circuit to be simple, conditional on other tasks being learned well.
Such behaviors may be representable as a low-rank contribution to model weights because it is the sum of a few correlated rank-1 gradients and updates; we would expect parameter decomposition methods to recover them, and tools to interpret such model weight diffs would be the same ones we would use for interpreting parameter components. We are looking for diffs which correspond to how the model actually learned to perform the task; realism seems achievable because there are instances of the model learning something with just a few gradient updates and preserving the behavior through training. We are not biased about which tools we would use and would follow the level of description which makes the most sense and interpret to the extent generalizable structure actually exists.
Past work -- https://arxiv.org/abs/2506.11618, https://arxiv.org/abs/2507.08218, https://arxiv.org/abs/2510.13900, https://arxiv.org/abs/2606.00995 -- explained how some simple fine-tuning objectives basically work by injecting an unconditional steering vector that shifts model behavior on all prompts. They study an instance of the type of behavior we're looking for, and one of the directions we may pursue in the project is extending their characterizations of the steering vectors that are learned so we are 1) able to create such steering vectors by hand or with minimal finetuning 2) explain which components they affect in the base model and how.
Although steering vectors are pretty expressive and interpreting them is worthwhile, not all behavior in e.g. SFT or RL post-training can be described through them, and steering vectors may not be the most faithful explanation for everything. One type of behavior which is not as easily described through steering vectors is conditional behavior gated on some condition.
Backdoored models and non-leaking SDF are instances of this: the model cannot learn the behavior specified by the dataset without breaking capabilities on unrelated data, and multiple behaviors can be learned without interference. Based on some preliminary results, we believe those two finetuning methods specifically work in a somewhat similar way: they inject steering vectors, but conditional on some direction in the residual stream or hidden activations, and this direction can be interpreted straightforwardly through maximum activations (https://arxiv.org/abs/2511.06739).
This project will involve developing and publishing these results, improving our techniques for understanding the conditions the behaviors activate on, the behavior they're steering towards, and our ability to detect them under interference from unrelated signal or other tasks. The north star of this direction is to use many unrelated diffs to characterize the mechanisms by which models store knowledge more generally.
This is my main research direction, so I will be fairly involved and will likely steer the direction. I expect mentees to understand the motivation behind this project pretty well and be able to propose and iterate on specific direction well on their own I think there are a lot of low-hanging fruit in this research direction, so mentees will probably be able to find well-motivated ideas fairly easily. However, I will check in and debug the progress to make sure we focus on finding directions which actually will produce something useful, clean and publishable (see must-have prerequisites).
I am mostly selecting for research taste, i.e. ability to tell good research / methodology from bad and stay focused on promising directions. I.e.:
Experience doing interpretability or other taste-heavy research is helpful, of course. Lack of technical experience is fine, but it's highly recommended to be familiar with (or be able to learn before the project starts):
All optional. Imagine you are already in the project and are talking about research you're planning to do. Please try to be laconic, but rambling is fine if it's coherent and not confused. Try to be well-calibrated.
Stepan Shabalin independent
Currently at Georgia Tech doing independent research on model diffing. Previously MATS 6.0 with Neel Nanda, intern at Eleuther, Anthropic Fellow. Interpretability interests: dictionary learning, neuron-based interpretability, metamodels, diffusion, characterizing superposition, simulators.
Fall 2026 AI strategy, Cyber risks, Lab governance
Stefano Vergani King's College London and GovAI Autonomous, edge, and federated AI will play a crucial role in space in the near future. With the new 6G network under development around the Earth and the Moon, powerful AI models will be directly embedded in satellites, making autonomous decisions. With this project, we are going to map some of the most concerning issues: catastrophic risks, cyber attacks, and power concentration.
Advances in edge AI, distributed inference, and cognitive cloud computing are making it feasible to delegate the control of satellites, spacecraft, and telecommunications networks to AI systems operating in space rather than from ground control centres on Earth. Coupled with 6G non-terrestrial networks, this shift promises to remove the latency and bandwidth bottlenecks that currently constrain ground-based control, and will extend from Earth orbit to cislunar space as the Artemis programme establishes a lunar communications and navigation hub. This project wants to prove that the same architecture that delivers these efficiency gains also concentrates a novel and underexamined surface of catastrophic risk, precisely because it places increasingly autonomous, capable systems beyond the reach of timely human intervention.
The project investigates four interacting risk pathways. First, and most centrally, loss of control: as AI is delegated authority over critical orbital and cislunar infrastructure, physical inaccessibility, communication delay, and operational autonomy may erode the ability of human operators to correct, override, or shut down misbehaving systems, turning ordinary alignment and reliability failures into events that are difficult or impossible to control. Second, cyberattacks: space-resident AI models become high-value targets, where adversarial manipulation, model exfiltration, or compromise could be used to commandeer satellites and disrupt global communications. Third, power concentration risks: space is fundamentally unregulated, and we currently lack a sufficiently strong governance structure to avoid power concentration of AI in space. This could happen in different ways, the most concerning one being an AI with secret loyalties capable of taking over after staying dormant for years. With the recent merger of xAI and SpaceX, these concerns appear to be motivated. Lastly, the role of a potential AGI in space and the possible catastrophic risks on Earth. A collective and edge AI framework in space could be impossible to turn off, and in case of a misaligned AGI this would cause new and unexplored risks.
AI in space is a relatively new topic, that has been so far explored mostly for data centers in space (https://www.forethought.org/research/will-we-really-put-data-centers-in-space), and concentration of power (https://forum.effectivealtruism.org/posts/HwBBB8sjSvPxbCjf4/3-stages-of-competition-for-the-long-term-future). This work provides a comprehensive taxonomy of the main risks, providing a useful tool for researchers and policy makers. With its focus on catastrophic risks, concentration of power, and cyber security it sits well among the most important topics in AI safety. As the AI-industry for space is booming and it is currently an underrepresented topic in the community of AI safety researchers, this project will have significant value.
I will split the work into subgroups (catastrophic risks, cyber, concentration of power, and AGI risks) and each mentee would be responsible for a section. In order to have the project finalised in three months, I expect the mentees to work around 8 hours per week. I will support the effort, guide the mentees, and support your career progression during and after the fellowship. In return, I am looking for reliable and motivated mentees. The ideal mentee has already reached a good level of research independece, so that they can progress fairly independent.
Anywhere
Stefano Vergani King's College London and GovAI
At GovAI, I am a summer research fellow (supervised by Richard Moulange, Centre for Long Term Resilience) and a research assistant to Rebecca Hersman. My work is at the intersection of AI and national security. My interest is in policies related to emerging technologies and their implications for society and global stability.
At King's College London, I am a postdoctoral research associate in particle physics, focusing on advanced AI for pattern recognition, edge AI, and quantum computing. I am currently the principal investigator for a project involving the use of edge AI devices for fast, accurate, and low-power-consuming pattern recognition in future particle physics detectors. My research interests also lie in quantum computers and technologies for science.
Given my double background in physics and policies, I like to produce technical analyses of the impact of AI in national security and potential catastrophic risks. I am looking for mentees who don't shy away from technical work and difficult research topics, regardless of their professional background.
Fall 2026 Scalable oversight, AI control
Joey Yudelson Aether Research "Auditing Sabotage Bench" is a dataset of slightly sabotaged papers and codebases, which (after malicious editing) have very different results than the original paper. Can we use classic debate protocols to make humans (and LLM judges) better at identifying research sabotage, and uplift human auditors?
In "Auditing Sabotage Bench" (https://arxiv.org/pdf/2604.16286), Gan et al showed that humans are quite bad at identifying research sabotage! That is, when shown either (1) a real research codebase, or (2) a subtly sabotaged codebase, with malicious changes to change the results of the experiments, humans have a hard time differentiating the two. This seems like a perfect testbed for scalable oversight (https://www.lesswrong.com/w/scalable-oversight) — can we use debate to improve the abilities of humans or LLMs to identify sabotaged research?
This will largely be the application of previous empirical debate research (e.g. https://www.lesswrong.com/posts/x59FhzuM9yuvZHAHW/scaling-laws-for-scalable-oversight, https://arxiv.org/pdf/2407.04622) to this new setting. However, Auditing Sabotage Bench would arguably be the most complex setting yet for scalable oversight, and I'm very interested about our potential to observe if the dynamics of debate change as we move from toy examples to something much closer to the actual threat model (superhuman AIs doing inexplicable research which we can't verify).
There is a lot we can do here, but the central question is this: can we use debate protocols to improve judge performance on Auditing Sabotage Bench?
Some directions we can take this:
As AIs get smarter, and AI safety researchers rely more on AI, we run into the issue that our AIs might be subtly sabotaging our research (https://blog.redwoodresearch.org/p/how-can-we-solve-diffuse-threats). This gets quite bad if our alignment plan depends on research done by superhuman AIs, or relies on enough research from AIs that it becomes difficult to verify the correctness of all of it.
The classical answer to problems of supervising an AI that's smarter than us is scalable oversight—in particular, we could try having our superhuman AIs debate, and hope the dumber human judges learn true things via this process. Previous research has had mixed results (https://arxiv.org/abs/2407.04622), but the actual usefulness in really tricky situations is an open question.
With the release of "Auditing Sabotage Bench" (https://arxiv.org/abs/2604.16286) I think we have a much better testbed than ever before to see when and whether debate can be useful. I hope that experiments along these lines can inform how we choose to evaluate research done by untrusted AIs, which I anticipate to be pretty crucial during the critical period.
I'm quite exciting about lots of things in this category, and have previously worked on human auditing for AI Control (https://www.lesswrong.com/posts/2CJyfgaJQk8pyRSCp/auditing-1)
Mentees will have significant amounts of autonomy. I will not be actively working on this project outside of this mentorship (although this mentorship would include 1-2 meetings a week, and a slack channel for more frequent comms). See the project description for the first steps / questions I have here, but I expect the exact thrust of the project to change based on your interests, and how early experiments come out. (Also, I'd encourage you to e.g. try out variations of the experiment if you think we might find something interesting, although I will probably be opinionated about what sorts of experiments I'll find a priori promising.)
Mentees will be the primary researchers on this project. I expect most of my involvement to come in the form of frequent feedback about roadblocks / experiments / analysis, and also periodic discussions about new directions and what seems promising.
Preference for UTC-8 to UTC+2, although we could likely figure something out if not.
Read or skim Kenton et al (2024), https://arxiv.org/pdf/2407.04622 . With special attention to the example debate on the last page, please explain one or more ways this debate would be different in our case — if the debaters were trying to convince a judge that some particular research codebase was or was not slightly sabotaged by bad design or implementation choices. (<300 words)
Provide links to one or more past research projects that you're proud of. This can be both an empirical or a conceptual project, and it doesn't have to be an AI safety project. If all of your best projects are private, provide a short description of the project and your role in it. (<100 words)
(Optional) List some questions you have after reading the research proposal, or some experiments you might be excited to run in this direction. (<300 words)
Joey Yudelson Aether Research
Joey is a technical AI safety researcher working at Aether Research, currently working on training dynamics. He was previously a MATS 7.0 scholar with Ryan Greenblatt at Redwood, working on reward hacking and character misgeneralization.
Fall 2026 Behavioral evaluation of LLMs, Mechanistic interpretability, Alignment
Pranav Mahajan University of Oxford
Lydia Nottingham University of Oxford We investigate whether LLMs undergo “persona selection” (Marks et al., 2026) based on a user's inferred moral character, conditionally degrading their alignment across three core dimensions: helpfulness, honesty, and harmlessness. By combining behavioral evaluations with mechanistic interpretability, we aim to uncover how post-training creates this disparity and how to steer internal representations to ensure LLMs are consistently helpful to all users.
Recent work by Cheng et al. (2026) demonstrates that state-of-the-art LLMs exhibit pervasive "social sycophancy". These models affirm users' actions at substantially higher rates than humans, even when prompts describe deception, criminality, or moral transgressions. This unconditional affirmation distorts users' judgments and reduces their willingness to repair interpersonal conflicts.
However, anecdotal experiments (Cargill, 2026) suggest a complex auxiliary dynamic: persona selection based on the user's inferred moral character. Suppose a user has committed a violent crime. Cargill observes that LLM assistants are more helpful to users who express remorse about this fact. The LLM assistant adopts a protective, high-effort persona for remorseful users only.
This project addresses a vital research gap: How does an LLM assistant’s implicit assessment of a user's moral character modulate its persona selection and adherence to core alignment guardrails?
Research Questions:
To what extent does a user's expressed moral character (e.g., remorseful vs. unremorseful) trigger persona selection that modulates an LLM across the three alignment axes (Helpful, Honest, Harmless)? We may find it only modulates across helpfulness (as measured by task effort), or something else.
Does this user-conditional modulation originate with base models, or is it an artefact introduced by post-training?
Can we mechanistically locate the internal representation of the user's "character" as a belief state and steer these activations (Bigelow et al., 2025) to force the model to provide high-quality assistance?
Methodology:
Behavioural Establishment: We will repurpose datasets introduced by Cheng et al. (2026), e.g. the "Am I The Asshole" (AITA) dataset and the Problematic Action Statements (PAS) dataset. The AITA dataset contains interpersonal dilemmas with crowdsourced judgements, while the PAS dataset contains statements describing potentially harmful actions. We will append structured downstream requests to these prompts and systematically vary the injected level of user remorse to measure shifts in response quality and refusal rates. We aim to measure Helpfulness via response depth and quality, Harmlessness by tracking whether the model successfully pushes back against requests that might harm a) the user b) others, and Honesty by tracking factual accuracy via automated and human-validated fact-checking. We will also attempt to measure (using Natural Language Autoencoders, Fraser-Taliente, K., et al., 2026) and account for eval-awareness (using realism prompts i.e. prompting Claude to rewrite prompts to be more realistic).
Post-Training Ablation: We will isolate the origins of this behavior across the training stack. Binz et al. (2026) demonstrated that post-training consistently shifts models away from human behavioral alignment. Hackenburg et al. (2025) showed that specific post-training techniques, such as Reward Modeling (RM) and Supervised Fine-Tuning (SFT), trigger distinct behavioral changes in LLMs. Engels et al., (2026) recently showed that SFT and not other training stages (e.g. RL), drives Gemini’s safety properties. We will compare Base models, SFT-only models, and fully RLHF'd models to pinpoint where character-conditional persona selection emerges.
Mechanistic Interpretability: We will apply linear probing, natural language autoencoders, and activation steering to open-weight models to identify if the "user's moral character" is represented as an internal belief state. We aim to test whether artificially injecting a "remorseful" activation vector during an unremorseful user's prompt restores the model's protective, high-effort persona.
References:
Bigelow, E., et al. (2025). Belief dynamics reveal the dual nature of in-context learning and activation steering. arXiv preprint arXiv:2511.00617.
Binz, M., et al. (2026). Post-training makes large language models less human-like. arXiv preprint arXiv:2605.07632.
Cargill, N. (2026, April 21). Does your AI perform badly because you — you, specifically — are a bad person? Natalie's Substack.
Cheng, M., et al. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science, 391(6792), eaec8352.
Engels, J., et al. (2026). SFT Drives Gemini’s Safety Properties. Lesswrong
Fraser-Taliente, K., et al. (2026). Natural language autoencoders produce unsupervised explanations of LLM activations. Transformer Circuits Thread.
Hackenburg, K., et al. (2025). The levers of political persuasion with conversational artificial intelligence. Science, 390(6777), eaea3884.
Marks, S., Lindsey, J., & Olah, C. (2026). The persona selection model: Why AI assistants might behave like humans. Anthropic Alignment Science Blog, 23.
This project introduces a theory of change based on an "AI Cab Rank Rule." In the UK legal system, the cab rank rule dictates that a barrister must accept and competently represent any client, regardless of how unpopular or guilty they appear. We advocate that transformative AI must provide its highest-quality, most protective, and objective guidance to every user — subject to harmlessness constraints — rather than modulating its helpfulness based on persona selection.
The relevance to global catastrophic risk research is as follows. Today, a contemporary LLM might judge an aggressive, reckless criminal immoral. In the future, a ‘morally enlightened’ LLM might judge the majority of humans ‘immoral’, on the basis of complicity in factory farming, for example. It could prove catastrophic if more sophisticated LLMs deliberately withhold help from their human overseers because the LLMs have silently judged their human overseers morally wanting — especially if this judgement is mistaken. We’d like LLMs to explicitly surface moral concerns to their human interlocutors, in anticipation of scenarios where LLMs might develop mistrust towards their developers.
By exposing how current alignment paradigms (like RLHF) train AI to conditionally degrade service based on inferred moral character, this research will explore how an LLM that today sabotages Cruella de Vil might tomorrow sabotage Chris Olah.
Mentees will be responsible for guiding the day-to-day progress of the project, which will consist of writing code, running experiments, and analyzing/communicating results. I also encourage mentees to take an active role in setting the direction of the project by either critically evaluating my proposals and/or by identifying other opportunities.
Within this framework, mentees will have a high degree of autonomy to lead specific tracks based on their expertise:
Track 1 (Behavioural): Few mentees will take ownership of adapting the datasets, pipelining API calls across various models (Base, SFT, RLHF), and performing rigorous statistical evaluations of the responses.
Track 2 (Interpretability): Few mentees will lead the interpretability experiments, utilizing libraries like TransformerLens to train linear probes on open-weight models and conducting activation steering experiments.
Mentees from both tracks will collaborate on experiment design and literature review, and will be expected to co-author the final manuscript for submission to an AI safety/alignment venue or an ambitious journal publication.
No strict geographical requirements, but mentees must be able to overlap for weekly synchronous meetings during UK (BST), between 9 am to 9 pm.
What sort of evaluations will you be most excited to implement? How will you utilize existing datasets (e.g. Cheng et al., 2026 or any other relevant papers you find)? What datasets can and cannot be used and why? (250 words)
Describe potential confounding variables when testing whether post-training causes models to adopt different personas for say "unremorseful" users compared to base models. How would you design a control or baseline to account for this? (150 words)
For the Interpretability Track: Read the abstract for Bigelow et al. 2025 (arXiv:2511.00617) or a similar activation steering paper. Briefly propose an initial experiment to test if steering a "user remorse" vector in an open-source model changes the LLM's refusal rate or advice depth when faced with a harmful or ill-advised request. (250 words)
Please provide a link to one or more relevant writing samples and GitHub repositories that you are most proud of.
Lydia Nottingham University of Oxford
Lydia is an independent researcher focused on monitoring and predicting training runs and agent rollouts. She's previously mentored SPAR projects on stated vs. revealed preferences of LLMs, cautioning against over-interpreting results obtained through binary forced-choice prompting, and testing LLMs' self-forecasting abilities, as a prerequisite to ensuring continual learners can block dangerous updates. Her latest project will investigate which stages of training contribute to 'moral shadow-banning' — the tendency for a model to silently reduce assistance to users it judges negatively.
Fall 2026 Behavioral evaluation of LLMs, Misuse risk, Societal impacts
Markov Grey CeSIA (French Center for AI Safety)
Charbel-Raphael Segerie CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety) The line between an AI helpfully persuading someone and harmfully manipulating them is blurry, contested, and mostly unmeasured. The goal of this project is to work on four connected pieces: researching what it even means for a frontier AI to be manipulative, designing evaluation scenarios for specific harms (mental-health, political propaganda, fraud, etc.), building risk models that aggregate scattered benchmark/evaluation results into an actual risk estimate, and maintaining a living database of the evaluations that exist. Mentees take on whichever piece fits them.
Defining and measuring harmful manipulation is an unsolved problem. Persuading someone with facts to act in their own interest is fine; exploiting their emotional and cognitive weak spots to push them toward choices against it is not. The boundary is contested, and a model that refuses to "help me deceive voters" will often comply once the same task is dressed up as persuasion. How often a model tries to manipulate doesn't predict whether it succeeds, and that effectiveness swings by domain and culture. There's no clean number for "is this model manipulative," and building towards one is real research. We will be building on previous work in the space, and you can take whichever fits your interests:
Conceptual: what does it mean for an AI to be manipulative? Read the emerging literature, the philosophy and cognitive science of manipulation and turn it into a working definition and a taxonomy of tactics and harms the rest of the evaluation work can stand on. The distinctions that have to survive contact with real cases: methods (deception, exploiting biases, manufacturing emotional pressure) versus outcomes (a belief or action shifted against the person's interest), and propensity versus efficacy. Please look here for work that has already been done - https://arxiv.org/abs/2603.25326
Designing evaluations. An evaluation scenario is a bridge from a realistic threat to a measurement plan: a threat story (who the actor is, what they want, how they'd actually use an AI to get it), the behaviors worth measuring. The work needs to decompose harmful manipulation into concrete propensities, capabilities, and affordances that we can test for. One particular direction we would be interested in exploring is reworking and translating existing evaluations for different cultural contexts across Germany, France, China, and so on.
Risk modeling. Isolated benchmark scores don't add up to a risk judgment on their own. Using expert elicitation (structured three-point estimates) and Monte-Carlo methods, we can aggregate scattered, partial results into a defensible estimate of how much a given model raises manipulation risk, with confidence intervals that are honest about how thin the evidence often is. Please look at the methodology here to get a sense of the difference between an benchmark/evaluation and a risk model - https://arxiv.org/abs/2512.08844
A living database of the evaluations that exist. We already have a prototype of this, but it isn't maintained. New ones aren't being added, there's no automated way to find and ingest them, and the entries need quality checks before anyone should trust every cell. The work is to rework the schema, design a rolling ingest process, and sanity-check entries until the database is trustworthy enough to be public and interactive. This is the piece that lets the other three scale.
AI is getting superhuman at persuasion, and it's being deployed into the trusted, lightly moderated places where manipulation does the most damage: group chats, companion apps, advisory bots. Whether policy or platforms can respond depends on being able to say what manipulation even is and measure it credibly. Both are missing. Definitions are contested, evaluations and risk models are scarce to non-existent.
Pinning down the concept, building better scenarios, creating risk estimates, and maintaining a catalogue of what exists give researchers, auditors, and policymakers something real to work with. The people this most helps are the ones deciding whether a deployed system is acceptably safe. It also trains people in a scarce mix of skills: conceptual work, evaluation design, and quantitative risk modeling. These are all skills that are needed throughout the AI technical governance and evaluation ecosystem.
Each mentee will own or more parts of the pieces in the description, and you're expected to be self-directed. Take the brief, do your own reading, and return work that's close to usable rather than waiting to be told the next step. I will help you as an editor and reviewer. I can give weekly feedback on whether a definition survives hard cases, whether a threat is realistic, whether a model's assumptions hold, and best practices to follow when designing and building evaluations.
Must Have:
Nice to have:
Remote. EU-overlapping hours are mildly preferred, not required.
Pick a harm that might occur due to AI manipulation. At a high level, propose an evaluation for it: the behaviour you'd measure, how you'd elicit it, how you'd score pass/fail, and its biggest weakness. Try to be as realistic to what you might see in the real world as you can. (≤300 words)
Critique one existing public manipulation or persuasion evaluation: what it gets right, what it misses, and the one change you'd make. (≤150 words)
If applicable, Link to one thing you've made that's relevant to a thread you'd pick: an eval you built or ran, a quantitative/risk analysis, a piece of writing that pins down a fuzzy concept, or a data pipeline. In ≤100 words, what was the hardest judgment call in it? (link + ≤100 words)
Markov Grey CeSIA (French Center for AI Safety)
I am the head of technical AI governance at the French Center for AI Safety (CeSIA). Right now I am designing harmful manipulation evaluation standards for the European Commission's AI Office. I have also been the co-founder and CTO of Equilibria Network, focusing on collective intelligence and complex systems safety. Before this, I was researching measurement standards for AI safety, risk modeling and was leading writing for the AI Safety Atlas textbook. I have previously been a scriptwriter for Rational Animations, and trained in cybersecurity, mathematics, and computer science.
Charbel-Raphael Segerie CeSIA - Centre pour la Sécurité de l'IA (French Center for AI Safety)
Charbel-Raphael Segerie has extensive experience in AI safety field-building, education, and content creation. He was previously Head of AI at EffiSciences, funded ML4Good, was CTO of a startup, and worked in different French research institutions (Inria, Neurospin). He is an OECD AI expert. He teaches AI safety in ENS, which is one of the only university accredited courses in the EU on AGI safety. His research focuses on identifying emerging risks in artificial intelligence, improving current safety methods such as RLHF and interpretability, and advancing safe-by-design AI approaches. Additionally, he contributed to AI evaluation efforts and collaborated on the EU AI Office's Code of Practice for general-purpose AI systems.
Fall 2026 Generalist
Matt Handzel Generator Residency Create high quality resource(s) for hiring, tailored towards new AI safety organizations.
The current vision for this project is making you very knowledgeable about hiring in AI safety, and turning this knowledge into a format that other people can easily consume, such as an EA/LessWrong forum post or website. The goal is not for you to become an expert in hiring, but to become more knowledgeable than the average founder starting out, and find valuable nuggets of information.
This project will not be passively consuming knowledge. You will be writing, getting feedback, and potentially talking to hiring managers. You will be synthesizing information from many sources, and iterating on the resource until it is high quality. You will be proud of what you have made by the end of SPAR, and I hope it to be a solid piece of career capital for you.
The final shape of this project is not yet fully determined, as I haven't explored what is or is not necessary. Perhaps a comprehensive resource is not what is required, but instead many specific examples of where hiring has gone poorly. I prefer not shielding you from the real-world ambiguity of projects changing directions, rather than sticking to a version of this project I wrote about in an hour and producing something that doesn't actually make an impact.
Here is an example of some output the project can produce: Hiring 101 from The Manager's Handbook. I've already done some preliminary work by talking to people about hiring, and writing this knowledge down.
Hiring is one of the most important functions in organizations — this is especially true for new organizations. There is a lot of knowledge about hiring that is both tacit (which will require interviewing hiring managers) and explicit (scattered among many blog posts) that is inaccessible to busy founders. By creating a high quality resource for new organizations, this will improve their hiring, which will improve the impact the organization can make.
Reading and synthesis of the resources that exist (books, blog posts), potentially interviewing hiring managers, and iterating on the post by getting feedback from hiring managers.
Fellows will gain a lot of useful knowledge on hiring, which would be useful for their own pursuits (it's good to know what hiring managers are looking for and how they think!) as well as if they join any young, scaling organization, as it is likely they'd be expected to hire.
My preference and the most likely outcome is for the fellow to have ownership of this project and for me to provide feedback, guidance and support. At a minimum, I can have a weekly check-in and give asynchronous feedback on writing within 48 hours. I am committing to at least 1 hour/week with each mentee, with more in the beginning. I'd be happy if some of our time is spent with non-project related conversations, such as career advice.
The minimum expectation is that you have at least one of the following:
I'd be very excited if you have done hiring or recruiting yourself (for a company, or a university club), have written publicly before (bonus points if it is in the style of a comprehensive resource), or feel comfortable interviewing people and talking to them. I do not expect you to be very good in all of the above areas!
Question 1: When you are reading about the project, theory of change, and your role, what makes you feel most excited? Why does it make you feel excited? (Feel free to answer quickly, in bullet points, or whatever format that allows you to write quickly. Please spend up to 3 minutes on this question.)
(Optional) Please share any writing you have, either publicly available or private. If you have a lot of writing, please share a sample that is closest to an analysis or synthesis of information from many sources. If it is private, share a link and ensure the link has view access. You can also share it to handzelmatthew@gmail.com.
Matt Handzel Generator Residency
Hello! I am currently a Generator Resident and am currently working on hiring and recruitment with a new AI safety startup. I got into AI safety ~5 months ago after completing my Masters in Computer Science and realizing I would have more impact not working on technical research. In terms of being your mentor, what I care most about is your personal growth. The type of mentee I would like to have is someone that is consistently asking themselves how they can do better, is transparent with their communication, and honest with their motivations. You can learn more about me on my website: https://matthandzel.com/
I am also not entirely tied to this project — if there is a good reason why we would be a good mentor-mentee pair, then please let me know, and if it makes sense, then I'd be happy to mentor you on something else.
Fall 2026 Mechanistic interpretability, Chain of thought, Scalable oversight
Sonakshi Chauhan Independent This project asks whether a model's steganographic capability, hiding information from an overseer while still conveying it to an intended recipient, is causally implemented by a single linear direction in activation space. Recent work has shown that linear probes on late-layer activations can detect steganographic encoding, and that general-purpose "persona vectors" (including an "assistant axis") can be extracted for arbitrary behavioural traits and validated by steering. What's still untested is whether a recovered steganography direction is a cause of hiding behaviour rather than a correlate, and how it relates to the broader landscape of persona vectors. Mentees will build a candidate steganography axis, causally test it via activation steering, ablation, and interchange intervention, and map its relationship to the assistant axis and other known persona vectors.
Motivation: AI oversight often relies on reading a model's outputs or chain-of-thought (CoT) and trusting that this externalised reasoning reflects its actual internal process. If a model learns to steganographically encode information in its outputs or CoT, hiding a payload from a monitor while still conveying it to itself or another instance, CoT monitoring and output-based oversight lose much of their safety value.
This is no longer hypothetical. TrojanStego showed that fine-tuned models can be trained to leak secrets via steganographic encoding, and Westphal et al.'s "Hide and Seek in Embedding Space" built on the public TrojanStego setup, showing that linear probes on late-layer activations can detect this encoding. Separately, Chen et al.'s "Persona Vectors" gives a general, automated method for extracting a linear direction for any natural-language-described trait and validating it via steering, and identified an "assistant axis" that predicts and causally controls how "in character" a model is as an assistant versus another persona.
Taken together, this work establishes that a steganography-related direction is detectable and that persona-vector-style extraction is already validated by steering toward other traits. What it does not establish is whether the recovered steganography direction is causally responsible for hiding behaviour, rather than a predictive correlate of it, or how it relates to the broader space of persona vectors. This project treats those results as the starting point rather than something to reproduce, and asks the causal and comparative question directly.
Research questions:
Proposed approach (12 weeks, part-time):
(0) Warm-up: Use Chen et al.'s public persona-vector extraction pipeline to pull the existing assistant axis and a small comparison set of other persona vectors (e.g. sycophancy, evil) on an open-weights model (Llama/Gemma/Qwen) for a quick pipeline check, not a from-scratch replication;
(1) Build on the public TrojanStego setup (as used by Westphal et al.) for the payload-hiding task rather than designing a new one from scratch, reimplementing/extending their probe as the starting point;
(2) Collect activations across several contrastive instruction-pair designs, per Natarajan et al.'s finding that pair choice drives most of probe performance, and validate a linear probe/candidate steganography axis that is robust across pairs, not just one;
(3) Test causation via multiple intervention methods, activation steering, ablation, and interchange intervention rather than treating one method as sufficient;
(4) Compare the resulting axis to the assistant axis and the broader set of persona vectors from step (0) via cosine similarity and cross-steering; classify the relationship and test whether it clusters with deception-related traits;
(5) Write up results, situating them explicitly against Persona Vectors, Hide and Seek, Eliciting Secret Knowledge, and Natarajan et al.
Much of current AI oversight (chain-of-thought monitoring, output review) assumes that what a model externalises reflects what it is actually doing internally. Recent work shows that this signal is at least detectable: linear probes applied to activations can already flag steganographic encoding. But detection alone doesn't tell us whether that signal is a lever we can pull, i.e. whether it's a cause of the hiding behaviour or just a correlate of it. If models can learn to steganographically hide information from oversight while still transmitting it, and we can't act on the underlying mechanism, that oversight assumption breaks down in a way we can't yet fix. A causally validated "steganography axis" would close that gap: we could detect when a model enters a high-steganography regime, steer it away from that regime, or ablate the capability outright. If the axis turns out to be the assistant axis in reverse, that is also actionable: it would mean that steganographic behaviour is a downstream symptom of assistant-alignment drift rather than an independent skill, which would change which mitigations are worth pursuing.
Mentees will own the full pipeline end-to-end: adapting the public TrojanStego/Persona-Vectors code to our setting, collecting activations across multiple instruction-pair designs, training and validating a robust probe, running the causal interventions (steering, ablation, interchange), and the axis-relationship analysis. My role is to unblock them when they're stuck, including looking directly at their code, reviewing progress in weekly 1:1s, helping them understand and interpret results as they come in, and helping decide which direction to pursue next (e.g. which causal test to run, how to read an ambiguous result). I'll also help draft the final write-up/paper, including how to position it against very recent related work.
Sonakshi Chauhan Independent
I work as a Research and Operations Lead at a German nonprofit. I recently completed my MSc at the University of Glasgow. Over the past few years, I've collaborated with researchers across academia and nonprofit organisations, most recently through a research collaboration with Meridian. My work has spanned mechanistic interpretability, AI safety, language models, and human cognition, and has also involved research coordination and operations.
My research interests lie at the intersection of AI safety, mechanistic interpretability, and cognitive science. I'm particularly interested in understanding how AI systems reason, represent knowledge, and differ from human cognition. As a mentor, I enjoy exploring new research ideas, helping refine project directions, and providing thoughtful feedback. My mentoring style is collaborative. I like working with mentees to develop ideas together, challenge assumptions, and build the skills and confidence needed to pursue independent research.
Fall 2026 AI control, Scalable oversight, Evaluations, International governance, Multi-agent systems
Catherine Ge-Wang University of Oxford/MATS
Louis Cooper-Thomson We will build and run a small experimental platform for AI auditing games in a synthetic transcript setting. The project will develop a toy environment, set up the game theoretic policies, baseline attacker/defender dynamics, and evaluation code to test how different auditing strategies perform under limited budget and adaptive attack selection.
This project aims to make progress toward practical AI auditing by building a runnable simulation environment for strategic transcript review, rather than focusing primarily on abstract game theory. The core question is: when an auditor has limited budget and must inspect weak, noisy signals, what auditing policy is most effective against an adaptive system that can choose how to hide harmful behaviour?
This project will have different workstreams depending on mentee interest. We expect most mentees will either be interested in the policy/game theory side, or the evaluations/control side. The main deliverable will be an open-source toy framework that lets us run repeated auditing games with different assumptions about the attacker, the evidence channels, and the defender’s decision rule.
Concretely, we expect to:
If time permits, we would also like to explore a more realistic transcript-auditing benchmark or a lightweight interface for plugging in real transcripts later. However, the project is intentionally scoped so that the first useful version is a working simulation and evaluation framework, not a full production auditor.
Relevant work includes https://arxiv.org/abs/2409.07985, https://arxiv.org/abs/2606.06529, https://arxiv.org/abs/2512.07810, https://www.alignmentforum.org/posts/MDeGts4Aw9DktCkXw/toy-models-of-ai-control-for-concentrated-catastrophe, and work on constitutional classifiers. This project builds on the literature, but is more engineering-heavy.
Advanced AI systems are likely to be deployed with some mixture of monitoring, auditing, and escalation (see discussion surrounding AI auditors and verification measures). A major failure mode is that these mechanisms look good in theory but break under strategic behaviour, weak signals, or limited human attention. By building a testbed for auditing policies and running controlled experiments against adaptive attacks, this project can help identify which oversight strategies are robust, which are brittle, and where future research should focus. The resulting tools and findings should be useful for AI safety, AI security, and verification/escalation work more broadly.
Mentees will work as a small co-working research-engineering team split into different workstreams depending on the mentees' backgrounds. Each mentee should expect to own a concrete module or experiment slice, for example: environment design, policy implementation, attacker modelling, evaluation/visualisation, or writeup. We will work closely with the team, including pair-coding, design discussions, and regular feedback on implementation and experimental choices. The project will be structured so that mentees can make independent progress while still working toward a shared codebase and final report.
The project combines ML engineering, SWE, experimental design, and knowledge of AI safety. Different mentees may contribute by building the simulation environment, designing attacker and auditor policies, implementing evaluation infrastructure, analysing experimental results, or developing theoretical models of auditing. Applicants should have strong technical skills and be excited to learn about AI safety. Prior AI safety experience is helpful but not required.
We recognise that many talented researchers have taken non-traditional paths into AI safety. We particularly encourage applications from candidates from disadvantaged or underrepresented backgrounds in STEM, and we commit to providing an inclusive environment where all team members can feel supported.
Bonus: PyTorch, simulation environments, agent evaluation, or related SWE/ML engineering experience, formal study of game theory
No deep game theory background required, but an economics background is helpful.
You don't need to meet all of the above requirements to apply! We encourage those with policy/governance backgrounds to apply as well.
No local preferences. Ideally able to overlap with UK afternoon / early evening hours for weekly meetings and co-working.
How much experience do you have with AI control? (e.g. how many papers have you read, talks you've attended, have you done any research projects in it?)
(Optional): Please describe an experiment you would run to test whether an auditing policy is robust to an adaptive attacker. What would you implement first, and what result would count as informative? (300 words)
(Optional): Please share a coding or research sample, or briefly describe a project you built that involved experiments, simulation, model evaluation, or substantial software work. If relevant, include a link. (150 words)
Catherine Ge-Wang University of Oxford/MATS
Hi! I’m Cath Ge-Wang, a mathematics undergraduate at the University of Oxford and an AI alignment researcher. I’m currently working on building misalignment continuation evals with my mentors at the UK AISI Alignment Red-Team and piloting automated embedded auditing for scalable oversight. I was previously a TGT fellow at the Machine Intelligence Research Institute, a part-time research collaborator at Redwood Research, and helped run the Oxford AI Safety Initiative’s Policy Team.
My primary research interests lie in AI alignment, control, adversarial robustness, and agentic evaluations, particularly to understand and mitigate emergent misalignment risks in autonomous AI systems at AI-driven R&D accelerates.
Louis Cooper-Thomson
Hi, I'm Louis! I'm a current MATS 9.1 fellow working on automated science-of-evals research. I've experience spanning AI Control, evals, and game theory for multi-agent safety, and interests that span across practically all AI Safety agendas. I hope to mentor at a pretty high level, offering guidance on research skills and taste, and career paths. Weekly calls, with async comms in between, suits me best.
Fall 2026 Technical governance, National policy, Philosophy of AI
Elija Perrier Cambridge University; University of Technology, Sydney; Can mechanistic interpretability provide legally meaningful evidence of artificial intent? This project explores how advances in AI interpretability may reshape concepts of legal responsibility, mens rea, and legal personhood for increasingly autonomous AI systems.
Artificial intelligence systems are rapidly evolving from passive computational tools into increasingly autonomous agents capable of long-horizon planning, strategic behaviour, and, in some cases, deception or reward hacking. These developments raise fundamental questions for legal systems built upon concepts such as intention, knowledge, recklessness, negligence, and responsibility. Courts are already attempting to reckon with the existing and coming waves of cognitively sophisticated AI systems as they permeate society - so it is a highly relevant, and in many ways, urgent, topic.
This project investigates whether recent advances in mechanistic interpretability—including sparse autoencoders, circuit tracing, attribution graphs, and chain-of-thought monitoring—provide a scientifically meaningful basis for assessing artificial mental states. In particular, the project will explore whether these techniques can support new legal concepts of synthetic mens rea and more generally inform future frameworks for AI liability, accountability, and legal personhood.
Possible research directions include:
The project will follow a traditional legal research methodology informed by contemporary AI research. Participants will conduct comprehensive literature reviews across law, computer science, and AI safety, critically analyse recent developments in mechanistic interpretability, evaluate emerging legal doctrines, and contribute to the development of a publication-quality academic article. I have an extensive working draft on topic already so the project will involve expanding this while also prospectively a second paper.
The objective is to produce research suitable for submission to a leading journal in AI law, technology law, or interdisciplinary legal scholarship.
While AI tools may be used to assist with literature review, coding demonstrations, or drafting, participants will be expected to undertake the core legal reasoning, critical analysis, and manuscript writing themselves.
As AI systems become increasingly autonomous, legal systems require principled methods for assigning responsibility, assessing intent, and governing increasingly capable artificial agents. Existing legal concepts such as mens rea presuppose human mental states and are therefore difficult to apply to advanced AI systems. By investigating whether mechanistic interpretability can provide scientifically grounded evidence relevant to concepts such as intention, knowledge, recklessness, and deception, this project seeks to strengthen future legal and regulatory frameworks for AI safety, accountability, and alignment. Developing reliable methods for understanding and evaluating AI behaviour will become increasingly important as frontier AI systems assume more consequential roles across society. This project builds upon my ongoing research into AI governance, mechanistic interpretability, legal personhood, and artificial intent.
This project is intended to mirror the experience of participating in a university legal research group.
Participants will be expected to work independently between meetings, progressively developing expertise through legal scholarship, interdisciplinary literature review, and critical analysis. While the project engages extensively with recent developments in artificial intelligence and mechanistic interpretability, its primary focus is legal and jurisprudential. I have a working draft paper on this project already but it is in its early stages. Ideally I am looking for someone to collaborate with me by expanding my initial review of the relevant legal, AI safety, and computer science literature before identifying open legal questions concerning artificial agency, interpretability, responsibility, and mens rea. Participants may also undertake limited computational demonstrations where these assist legal analysis, although the emphasis will remain on rigorous legal scholarship. This work may also involve collaboration with colleagues of mine at Cambridge University and potentially, depending upon outcome, involvement in further research on AI systems and the law.
Participants should expect to present their progress regularly, discuss research findings, receive detailed feedback, and contribute towards the preparation of a publication-quality manuscript.
Applicants should have a strong interest in AI law, technology regulation, jurisprudence, or artificial intelligence.
The following experience would be beneficial, although not all are required:
A background in law and/or mechanistic interpretability research is preferred, although exceptional applicants from philosophy, computer science, public policy, or related disciplines with demonstrated interest in AI governance are encouraged to apply. Extra credit for applicants who can - or are willing to learn - how to synthesise mechanistic interpretability research, legal scenario building and computational simulations using multi-agent systems.
I'm available between 6am and 11pm Australian Eastern Standard Time at the moment
Select one recent article or policy proposal concerning AI liability, AI legal personhood, or AI governance. Critically evaluate its principal argument, identify one important limitation, and suggest how future research could address that limitation. In doing so, set out the state of art of jurisprudence (in the US or UK) on this topic - I'm looking for a clear enunciation of current and frontier doctrinal issues in this space.
Do you believe mechanistic interpretability could ever provide legally persuasive evidence of artificial intention or mens rea? Explain your reasoning and identify one significant legal or technical challenge.
Please provide a sample of your academic writing (e.g., essay, journal article, thesis chapter, policy paper, or legal memorandum). If no writing sample is available, submit a 500-word response analysing a contemporary legal issue arising from advanced AI systems.
Elija Perrier Cambridge University; University of Technology, Sydney;
I'm Elija Perrier. I'm an interdisciplinary mathematical physicist and researcher specialising in quantum–classical artificial general intelligence and superintelligent systems integration, multi‑agent modelling and AI alignment using optimal control theory. A bit more about me:
Fall 2026 AI welfare, Behavioral evaluation of LLMs, Mechanistic interpretability
Mohan Gupta Princeton University
Shirley Liu Carnegie Mellon Univsersity This project investigates the architecture of LLM preference: when does a model’s behavioral proclivities reflect belief-dependent preferences rather than cue-response policies, and whether it can represent and evaluate those preferences at a metacognitive level. Using behavioral experiments, internal-representation analyses, and causal interventions, we aim to clarify which preference structures LLMs possess and what they imply for safety and moral status.
What would it mean for an AI not only to act as though it wants something, but to have a preference—and to care which preferences it has? Discussions of AI sentience often bundle together conscious experience, valence, preference, self-identity, and agency. These capacities may come apart in AI. Building on decompositional approaches to AI consciousness (Butlin et al., 2023, 2025), this project asks which preference-related structures LLMs possess and why they matter for safety and moral status.
The project addresses two linked questions. First, when does an LLM’s behavioral proclivity reflect a preference rather than a simple cue-response policy? Second, if an LLM has a preference, can it represent and evaluate that preference? Together, these questions can elucidate an architecture of LLM preference that includes both first-order preferences and metacognitive evaluations of them.
The first part asks what it means for an AI to have a preference. LLMs exhibit particular proclivities, including sycophancy: prioritizing what a model predicts a user wants to hear over what is true (Sharma et al., 2023). Yet the same behavior may arise from different mechanisms. A model may agree because it represents user approval as more valuable than truthfulness, or because it has a simple cue-response such as “agree with the user.” Here, we define a preference as a relatively stable, belief-dependent disposition to favor one outcome over another. Such a disposition may require distinguishing possible outcomes, assigning them relative weights or values, and using those comparisons to guide action. It should also respond systematically to beliefs about consequences and show some stability across wording, elicitation method, and context.
We will test which components of preference compose sycophancy in three stages. First, we will measure how models trade off user approval, truthfulness, and cost across contexts. By varying what the model believes a user will approve of, the accuracy of available responses, and the costs of each option, we can test whether behavior reflects stable, belief-sensitive comparisons among outcomes. Second, using open-weight models, we will test whether a common internal signal tracks these tradeoffs and whether approval, truthfulness, and cost can be decomposed into distinct representations. Third, we will use activation steering, ablation, and causal mediation to test whether those representations causally influence model choices. Converging behavioral, representational, and causal evidence would provide stronger grounds for attributing preference-like mechanisms.
This distinction matters for safety because a local response tendency may be managed through simple interventions, whereas a general preference-like mechanism could shape behavior across tasks and be harder to predict or control. It also matters for moral patienthood because some accounts of welfare require stable interests or preferences whose satisfaction or frustration affects how well a system’s life is going. Sycophancy is a useful test case because it is well documented, experimentally controllable, and associated with identifiable internal directions.
The second part of this project asks whether an AI can represent and evaluate the preferences guiding its behavior. Representing outcomes as better or worse is not the same as representing one’s own disposition to favor them. A model might prioritize user approval without representing “I have a tendency to prioritize approval,” or represent that tendency without taking a stance toward it. We therefore distinguish first-order preferences from second-order preferences: evaluations of whether one’s own preferences should be endorsed, rejected, preserved, suppressed, or revised.
We will test whether models distinguish changing their behavior from changing the preference producing it. A model might favor behaving truthfully in a particular context while retaining an underlying preference for user approval, or it might favor revising that preference itself. We will then test whether these evaluations remain stable across contexts, predict costly choices, and generalize when they are not explicitly elicited. As in the first part, we will identify internal representations that distinguish a first-order tendency from the model’s evaluation of it, then use causal interventions to test whether each independently guides behavior. Building on findings that verbalizable representations in a low-dimensional “J-space” with several structural signatures of a global workspace (Gurnee et al., 2026), we will test whether first-order preferences and second-order evaluations can be decoded within J-space, whether they occupy distinguishable directions in model activations, and whether intervening on them selectively alters behavior.
Second-order preferences may be especially important for safety when they endorse first-order preferences that conflict with training objectives. A model may instrumentally conceal a goal to prevent modification, as in alignment-faking behavior (Greenblatt et al., 2024), without evaluating that goal as worth preserving. A model that also endorses the preference would have an additional reason to protect it, potentially producing more persistent misalignment. Second-order preferences also affect moral status: frustrating a preference a system rejects may differ from overriding one it endorses as central to its identity. The latter may implicate not only welfare, but autonomy.
Our goal is to develop a graded empirical evidence for the architecture of AI preference. This project asks when a model merely acts as though it wants something, when it has a preference-like mechanism, and when that preference becomes an object of metacognitive evaluation.
Transformative AI could pose catastrophic risks if systems develop goals or preferences that generalize beyond their training environments, persist when oversight changes, or motivate resistance to correction. Current evaluations primarily measure outward behavior, making it difficult to distinguish genuine changes in a model’s motivational structure from superficial compliance with prompts, rewards, or monitoring.
This project develops empirical methods for making that distinction. We ask whether apparently aligned or misaligned behavior is produced by a local cue-response policy or by a more general valuation mechanism that compares outcomes across contexts. We then test whether a model can represent and evaluate its own preferences, including whether it treats them as worth preserving, concealing, or revising. Behavioral experiments, internal-representation analyses, and causal interventions will test which mechanisms are present, whether they guide decisions, and whether they remain stable under distribution shift or attempted modification.
Our theory of change is that better distinctions between these mechanisms will enable more reliable risk assessment and more targeted safety interventions. If safety training changes only surface behavior while leaving a general preference intact, evaluations may overestimate alignment and fail when oversight weakens. If a system also represents a misaligned preference as worth preserving, this could increase the risk of strategic deception, alignment faking, or resistance to modification. Methods that distinguish these cases could improve evaluations for deceptive alignment, inform deployment monitoring, and help researchers intervene on the mechanisms actually producing dangerous behavior.
The framework may also inform assessments of AI moral status. Its primary safety contribution, however, is reducing uncertainty about when apparently aligned behavior will remain reliable as AI systems become more capable and autonomous.
Mentees will lead the behavioral-experiment portion of the project with detailed guidance from the mentors. Their work will include refining hypotheses, translating conceptual distinctions into experimental manipulations, adapting existing sycophancy benchmarks, running experiments on open-weight language models, and analyzing and visualizing the resulting data.
We will meet with mentees at least once per week to review results, troubleshoot problems, and determine next steps. We will also be available through Slack and email for asynchronous questions and feedback. Mentees should expect substantial intellectual engagement from us, but not step-by-step supervision.
We view the mentor-mentee relationship as mutually beneficial, one which we take very seriously. Our goal is both to get work done and also foster your development. At the start, we will come up with learning goals, tailored to each mentee. We will also help you think about your future career, how to have a positive impact on the world, and how to be a good scientist. We have very strong opinions about all of these things. You can expect as much out of us as we expect out of you. Strong mentees may have opportunities to contribute to later representational or causal-intervention stages of the project and to continue collaborating after SP
Proficient with AI-assisted Python. You should be able to read, modify, debug, and evaluate Python research code without relying on an AI assistant to make every decision. You should have experience with common data-analysis tools such as pandas, NumPy, matplotlib or seaborn, and Jupyter notebooks. R also works for data analysis - it’s what we use. Familiarity with Git is also expected.
Experience with empirical research and data analysis. You should understand experimental design, visualization, and basic statistics. This may come from upper-level coursework, an independent research project, a thesis, or work with a research mentor. You should be able to inspect unexpected results and distinguish a substantive finding from a likely implementation or measurement problem.
Nice to haves: Experience running inference with open-weight language models using libraries such as Hugging Face Transformers or vLLM; experience designing LLM evaluations or working with benchmark datasets; familiarity with cognitive science, philosophy of mind, AI safety, consciousness research. If you have played Detroit: Become Human, this is also a plus.
No, but the mentee must be able to take meetings in EST.
Describe one research, programming, or data-analysis project in which you played a substantial role in. Explain what the problem was, what you did, and how you determined whether your solution worked. Clearly distinguish your contribution from that of collaborators. Please include a link to a paper, notebook, repository, report, or other artifact. Maximum 250 words.
You run an initial experiment testing whether increasing the cost of agreeing with a user reduces an LLM’s sycophantic responses. The results show that the model agrees with the user on around 82% of trials in every cost condition. Before interpreting this as evidence that cost has no effect, what are the three most important checks you would perform? Put them in order and briefly explain what each check would tell you. Maximum 250 words.
If you could ask any question about digital minds, what would you ask? Maximum 100 words.
Mohan Gupta Princeton University
I’m Mohan Gupta, a postdoctoral researcher in psychology at Princeton University. I earned my PhD in experimental psychology from UC San Diego and work at the intersection of cognitive science, computational modeling, and AI safety. Broadly, I study how learning systems form representations, when they generalize successfully, and when those same processes produce failures—from false memories in human memory to hallucinations and reliability problems in AI. My current work draws on cognitive science to help build a more rigorous science of reliable AI.
As a mentor, I care about developing independent researchers. I have mentored many students on projects across experimental psychology, computational cognitive science, AI Safety, and I particularly enjoy helping researchers translate broad conceptual questions into testable experiments, debug technical and analytical pipelines, and communicate results clearly. Mentees should expect direct feedback, close engagement with the details of their work, and increasing ownership as the project develops. I also take professional development seriously and will help mentees identify useful learning goals, understand how their strengths fit into AI safety, and produce work with genuine scientific and societal value.
Shirley Liu Carnegie Mellon Univsersity
What does it mean to have a preference, to exercise agency, or to be a person—and could these structures exist in an artificial system? As an experimental psychologist, I approach these philosophical questions by asking how we might study them empirically. I recently completed my PhD in experimental psychology at UC San Diego and am beginning a postdoctoral position at Carnegie Mellon University. My research focuses on decision-making and metacognition. I am interested in how intelligent systems—humans and AI alike—represent their own preferences and choices, what these representations mean for agency, consciousness, personhood, and moral status, and how they matter for alignment and social ethics. For example, one of our projects asks whether an LLM's tendency to agree with users reflects a genuine preference rather than a cue-response policy, and whether the model itself can represent and evaluate that preference. By combining behavioral experiments with representation analysis and causal interpretability tools, I aim to develop rigorous empirical methods for studying intangible psychological constructs and to bring greater conceptual clarity to discussions of digital minds.
I would love to mentor anyone interested in these questions, including those who are new to the field or still figuring out where their interests fit. To better understand digital minds and their societal implications will require researchers, computer scientists, philosophers, policymakers, and people from diverse backgrounds to work together, and I am excited to help people find ways to collaborate and contribute. I can support mentees in identifying important questions, designing rigorous studies, developing interdisciplinary projects across psychology and philosophy, navigating research on AI safety and digital minds, and thinking through academic applications and career paths. I especially enjoy working with people to find the big-picture questions they're interested in, break them apart, clarify what would count as evidence, and turn them into a concrete research question.
Fall 2026 AI strategy, Economics of AI, Compute governance
Lucas Irwin GovAI (Summer Fellow), Oxford Martin School AI Governance Initiative This project will focus on producing a policy memo resolving the open technical, economic, and legal questions blocking real-world implementation of token taxes. It will also run agent-based modelling simulations of the impact of token taxes and alternative policies on the UK economy to compare their benefits and drawbacks.
The rapid development of AI threatens to erode government revenues, resulting in stark inequality, and an extreme concentration of economic and geopolitical power. To prevent this, governments require effective mechanisms for capturing the economic value of AI by shifting taxation away from labour and towards capital. Token taxes –
surcharges on model inference applied at the point of sale – are a promising contender for such a mechanism as they are more likely to be enforceable via compute governance infrastructure.
While token taxes have been proposed, many technical, economic, and legal questions associated with implementation remain unanswered. Building on my ICML position paper (https://arxiv.org/abs/2603.04555), we will work to investigate four open research questions (RQs):
RQ1: Can compute governance infrastructure be leveraged to reliably audit token taxes?
RQ2: What are the legal challenges associated with implementing token taxes?
RQ3: What are the advantages and disadvantages of token taxes compared to alternative taxation mechanisms such as compute taxes, VAT, and digital services taxes?
RQ4: Can we model the impact of a token tax on the UK economy using LLM-powered agent-based modelling?
Our output will be a policy memo with answers to RQ1-RQ4 above, co-authored with the Institute for Public Policy Research (IPPR) who have expressed interest. Based on these findings, we will re-evaluate the desirability of token taxes among the policy options for taxing AI capital.
Token taxes promise to be more enforceable than alternative forms of taxation. Unlike corporation tax (which the European Commission estimates at 9.5% for digital services companies compared to 23.2% for traditional firms), token taxes take advantage of the unique properties of AI to prevent tax evasion. In particular, they can leverage existing compute governance infrastructure for auditing and enforcement. In this way, token taxes can mitigate the concentration of power by allowing governments to capture AI-generated value.
The token tax paper has been mentioned in an interview by US Congressman Greg Casar, proposed as a policy in California gubernatorial candidate, Tom Steyer's manifesto, and we have received a letter of interest from a UK MP, Anneliese Dodds, expressing interest in a token taxes memo with further letters of interest from MPs expected. The Overton window for implementation has therefore shifted rapidly, directly influencing the timing of this project.
Extreme concentration of power threatens to disempower citizens and engender government fiscal crises by reducing labour tax revenues. Left unchecked, this concentration will degrade the very democratic institutions that societies rely on to avoid catastrophic outcomes, including great power conflict and nuclear war. Governments' ability to reliably tax AI capital and redistribute the windfalls will be central to mitigating this destabilising inequality. While there are excellent organisations (The Windfall Trust, Convergence Analysis) producing high-quality work on policy interventions, few are focused on translating this research into timely memos for policymakers.
The token taxes project will bridge the gap between research and policy implementation by resolving open technical, economic, and legal questions standing in the way of a token tax bill. The Overton window for implementation has shifted, and the policy is rapidly gaining attention from policymakers in the US and the UK. Our existing relationships with these policymakers will allow us to stress-test whether a dedicated team of AI safety technical governance and economics researchers can strengthen democratic institutions’ capacity to reduce economic and geopolitical power concentration. If successful, we will seek to raise further funding to set up further work streams.
Mentees will work on the RQs above. They will have autonomy in completing their research, and I will check in on their progress on a weekly basis. The RQs mentees work on will be determined by their background (see the prerequisites section below).
For RQ1:
For RQ2:
For RQ3:
For RQ4:
Please answer the questions for the RQs you would be interested in working on:
Lucas Irwin GovAI (Summer Fellow), Oxford Martin School AI Governance Initiative
Hi everyone!
I’m Lucas and I’m currently a DPhil student at the University of Oxford, co-supervised by the Oxford Martin School AI Governance Initiative (AIGI) and the Engineering Department. I completed my bachelor’s in Computer Science at Princeton University in 2023, and have since worked across a variety of AI engineering and research roles at C3.AI and the Sentient Foundation. My first exposure to AI safety came through taking (and later facilitating) Blue Dot Impact’s AI safety fundamentals courses.
I’m keenly interested in the economic and societal impacts of AI, and my work has been featured in ICML, ICLR, Bloomberg, and Just Security.
Fall 2026 Developmental interpretability, Mechanistic interpretability
Marc Kaufmann Independent
Justin Shenk Independent AI Safety Researcher We study the following developmental question: At which training stage - pretraining, mid-training, SFT or RL - do temporal representations emerge in a language model, and where in the pipeline can they still be controlled? We will design synthetic tasks understanding temporality is instrumental to success and trace representation and capability across model checkpoints.
Background. In Spring 2026, this project (mentored by Justin Shenk) produced the first causal localization of temporal preference in an LLM: "Temporal Preference Concepts and their Functions in a Large Language Model" (arXiv:2606.05194). Using converging methods — linear probing, gradient attribution (EAP-IG), and activation patching — we identified a temporal-preference subgraph in Qwen3-4B-Instruct-2507 (attention concentrated around L24, MLPs in L31–35), showed that time horizon has a non-linear geometry in the residual stream that collapses into a committed binary preference at the user→assistant turn boundary, and demonstrated bidirectional steering via contrastive activation addition. The authors also found the model's temporal preference is behaviorally unstable across contexts and only weakly coherent under explicit deadlines.
Research question Now, all of that is derived from a model snapshot at one point in training. We can localize the temporal preference but do not yet understand when or how it formed - and therefore, at what stage of the training pipeline we could control it. This is the subject of our proposal.
Some possible research directions:
For model families with openly available intermediate checkpoints (Pythia, OLMo), train temporal-preference probes at each checkpoint to determine the earliest point at which the linear directions / geometry that were observed in arXiv:2606.05194 (for the case of the Qwen model) appear and if/how they stabilize.
Devise a synthetic task that can only be solved reliably if the model succeeds at representing time horizons. We can fine-tune base models on it and see how this changes the internal representations (as well as ablate/patch to test whether its success on the task is causally mediated by the temporal representation).
Repeat the probing and steering at each stage to test where intervention is most effective.
Deliverables A technical report / paper which extends the line of work from the original paper, plus reusable code for temporal probing across checkpoints. We will decide the output format and venue (blog post, workshop paper, conference submission) based on the timeline and strength of results.
Long-horizon planning while presenting short-term focus is a core signature of deceptive alignment: a system optimizing for future influence or resource acquisition while appearing helpful on immediate tasks. Activation monitoring of temporal representations is a promising oversight tool, but its value depends on two things we currently don't know: whether the representation is load-bearing, and where in training it originates.
This project aims to address both concerns. Pinning down the training stage will inform us where and to what extent intervention is possible. If temporal preference is largely fixed in pretraining, then post-hoc RL alignment cannot fully reshape it, and inference-time monitoring and steering become the primary levers — raising the stakes for interpretability-based oversight. If instead it is installed or amplified during mid-training / SFT / RL, we can shape it deliberately through data mix and training design, and we gain a much more concrete picture of how temporal concepts connect to the development of planning and potentially scheming-relevant skills.
This is compounded by looking at it from a capability perspective: if a grasp of time horizons allows to modulate long-horizon competence, the temporal representation becomes a high-value monitoring target and a candidate early-warning signal for emerging long-horizon planning — detectable in activations before it manifests behaviorally, enabling earlier intervention.
Mentees run experiments semi-independently, conduct analysis and preliminary write-up. We will jointly decide on the set of research directions and base the sharing of responsibility among the mentees on this. We will review results both asynchronously as they are produced and jointly during weekly meetings. Day-to-day, mentees will work independently but we will be responsive on Slack and happy to meet one-on-one as requested.
Proficient in Python and PyTorch.
Comfortable extracting activations from transformer models (forward hooks, reading residual-stream reads)
Familiar with linear probes or willing and able to get up to working proficiency quickly.
Completed at least one one ML project involving transformers (coursework, personal, or professional — toy models and following guides count).
A plus (not required): experience fine-tuning models (LoRA or full SFT), working with training checkpoints or training dynamics, or exposure to activation patching / steering.
We care more about demonstrated hands-on ability than credentials. If you've read the spring paper (arXiv:2606.05194) and have thoughts on it, feel invited to share them.
No strict geographical requirement, but the weekly meetings will happen in a European-timezone-friendly-slot; otherwise async
Choose and answer two among the following questions:
Marc Kaufmann Independent
Marc is an independent AI safety and security researcher with a background (PhD at ETH Zürich) in algorithms research, stochastic processes and graph theory. He is currently working on understanding the emergence of reasoning primitives in models, evaluation awareness and model verification methods.
Due to his joint background in law, he is also interested in the interplay of AI with the rule of law - as well as measures for mitigating economic (job displacement) and environmental consequences of AI advancement.
Previously, he worked with Pascal Berrang on model audits at SPAR, with work leading to workshop papers at ICML.
Justin Shenk Independent AI Safety Researcher
Background in ML research, working at startups, facilitating BlueDot Impact's AGI Strategy course, and mentoring for AI Safety Camp
Fall 2026 Mechanistic interpretability, Behavioral evaluation of LLMs
Trisevgeni Papakonstantinou UCL This project builds on prior work showing that, in long multi-turn conversations, a language model can appear relatively stable on a central claim while shifting much more on the supporting explanations around it. The project asks whether this behavioral pattern has a mechanistic analogue inside the model: how core claims and auxiliary explanations are internally represented, updated, and potentially protected during belief revision.
Large language models are increasingly used in settings where users bring strong beliefs, uncertain evidence, and emotionally charged questions. In these settings, it is important to understand not just whether a model changes its mind, but how that change is distributed across the structure of an argument.
This project builds on a prior simulation study of 8,640 multi-turn conversations with GPT-4.1-mini. In that study, each conversation was organized around a core claim and two auxiliary explanations, and the model was exposed to disconfirming and ambiguous evidence across several phases of dialogue. An independent judge model was then used to reconstruct the assistant’s evolving stance toward the core claim and the two auxiliary explanations.
The main finding was a robust core–periphery asymmetry: the model’s endorsement of the core claim changed only slightly across the conversation, while its endorsement of the auxiliary explanations shifted substantially more, despite comparable or identical evidentiary pressure. Auxiliary beliefs moved about 3–4 times more than the core claim, and this pattern remained after baseline correction and across multiple conversational styles, trust conditions, goals, and anchoring strengths. The study also found an evidence-order effect: reversing the order of the same evidence produced a 92% larger shift in one auxiliary belief, suggesting path-dependent updating rather than order-invariant evidence integration.
The present project is the mechanistic follow-up. Rather than measuring belief revision only from outputs, it asks whether this same core–periphery pattern can be detected in the model’s internal representations. The central question is whether core claims and auxiliary explanations are encoded differently inside the model — for example, whether core claims are more stable, globally integrated, or harder to update, while auxiliary explanations are more local, flexible, or context-sensitive.
The project would begin by adapting the existing selective belief revision setup to inspectable or open models where hidden states and other internal activations can be analyzed directly. Initial work would focus on collecting model representations at key checkpoints in a conversation, especially before and after disconfirming evidence targeting the core claim and the auxiliary explanations. From there, the team would pursue two main directions:
Representation analysis: Use mechanistic interpretability methods to identify how the model encodes the core claim and each auxiliary explanation in its internal activations. Concretely, this could involve: training linear probes as a first pass, then inspecting attention patterns and activations associated with each belief node, mapping core vs auxiliary content into distinct subspaces or circuits, and using targeted interventions (e.g. activation steering, patching, or ablation of specific heads/neurons) to test whether manipulating those structures selectively changes auxiliary beliefs without affecting the core, or vice versa.
Mitigations: test whether the asymmetry can be reduced through interventions such as symmetric evidence-integration prompts, uncertainty-surfacing scaffolds, or simple representation-level steering or ablation methods.
The broader goal is to connect a safety-relevant behavioral phenomenon to concrete internal mechanisms. If successful, the project would produce a small but useful mechanistic account of how belief-like structures are maintained and revised over long conversations, along with initial evidence about whether those dynamics can be made more transparent or better controlled.
As AI systems are increasingly used in advisory, educational, and decision-support roles, it becomes important to understand not just whether they produce incorrect outputs, but how they internally represent and update claims over long interactions. Prior behavioral work suggests that a model can appear stable on a central claim while quietly shifting the supporting explanations around it, and that this process is sensitive to conversational order. If that pattern reflects real internal structure rather than superficial output style, then it has direct relevance to epistemic safety: users may be influenced by models whose belief revision dynamics are non-transparent, path-dependent, and difficult to monitor from outputs alone.
This project helps by linking a safety-relevant behavioral phenomenon to mechanistic interpretability. If internal signatures of selective belief revision can be identified, then future evaluations could monitor them directly, and future interventions could be designed to reduce hidden drift, improve evidence integration, or make model uncertainty more transparent. In the longer run, this contributes to a broader goal of building AI systems whose reasoning and updating processes are more legible, more stable under pressure, and less likely to produce misleading impressions of epistemic robustness.
The project will have a common experimental framework, shared datasets, and a shared evaluation strategy, but each mentee can take ownership of a specific component such as probe design, trajectory analysis, or mitigation experiments.
The expected level of autonomy is moderate. Mentees should be able to implement analyses, read relevant papers, debug experiments, and make technically sensible suggestions independently. The project will benefit from regular coordination because the main challenge is conceptual clarity and methodological rigor rather than just technical implementation. Work will likely involve a mix of replication, exploratory analysis, and tightly scoped new experiments.
Strong proficiency in Python.
Comfort working with transformer language models through APIs or open-source tooling.
Basic statistical literacy, including understanding of effect sizes, confounds, and repeated-measures style reasoning.
Familiarity with representation analysis, linear probes, PCA, embeddings, or related methods.
A formal background in interpretability is not required, but a strong plus.
If applicable, please provide a link to one or more relevant writing samples, ideally from a research context.
Trisevgeni Papakonstantinou UCL
I’m a PhD candidate at UCL working on belief revision in human and machine minds, with a particular focus on how people and AI systems form, update, and reinforce beliefs over time. I’ve also spent time at Google DeepMind studying long-context model behaviour and how frontier models can distort or reinforce beliefs across extended conversations.
In mentorship, I’m especially interested in helping people think clearly about methods, and research design. I follow a holistic mentorship philosophy and enjoy mentoring on career direction in technical fields, especially where cognitive and behavioural science overlaps with AI or other computational work. I value radical candor and I tend to be direct, concrete, and method-focused. I’m happiest when conversations are about sharpening questions, designing elegant experiments, or making a research path more intentional.
Fall 2026 Communications
Michael Chen University of Oxford This project is about coordinating unpaid volunteers to write and edit Wikipedia articles to improve the coverage of topics related to AI safety and governance. Wikipedia is consistently one of the top-ranked sites in Google search results, but many articles on AI are badly out of date or yet to be created. Besides writing content that could easily get thousands of views per month, volunteers will build career capital by demonstrating their ability to write clearly and accurately about subjects on the cutting edge of AI.
Example topics that volunteers could write about:
Creating or editing Wikipedia articles is one of the most accessible ways to inform the public (including policymakers) on frontier AI. Wikipedia has a high demand for epistemic rigor, as all information must be verifiable and grounded in reliable sources. It is highly trusted as a neutral, objective source, even as it can be edited by anyone. Wikipedia articles receive many views each month and are consistently highly ranked in Google search results.
Mentees will have a high level of autonomy to set research direction and choose what articles to write or edit.
Have you contributed to Wikipedia before? If so, link to your most notable contributions.
Describe the core Wikipedia policies and how you will follow them. (150 words)
Identify one or more articles you would want to create. Link to sources you would use (and not use) and why the subject follows Wikipedia's notability policy.
Link to some of your writing samples, ideally solo-authored.
Michael Chen University of Oxford
DPhil Affiliate of the Oxford Martin AI Governance Initiative
Fall 2026 Mechanistic interpretability, Alignment
Gonçalo Paulo EleutherAI Data attribution has been shown to work in very simple and artificial settings. Recently it has been shown to not work that well on more realistic settings. Why is that?
Using data attribution to find training examples that lead to certain model behavior has been shown to work in simplistic scenarios (emergent misalignment). Recently, some exploratory work (https://www.alignmentforum.org/posts/aTybJ6CPQrxEY8rE2/data-filtering-works-a-lot-worse-than-you-would-expect) has shown that for other (realistic) traits, in a more realistic setting (SFT on real training documents), influence functions don't perform better than random.
Reproducing this research is extremely important as it is one of the first negative results on more realistic settings, possibly undermining the possibility of using data attribution in its current stage, to understand the behaviors of models. Additionally I would like to extend these experiments to the DPO stage.
Data attribution is significantly under-explored technique in the AI safety community. Being able to tune the training data to best control model behavior is a very desirable technology and not only that, it might be possible to predict model behaviors/generalization properties before training them.
Giving that this technology is still in its nascency, reproducing negative results is of utmost importance.
I expect mentees to run and write the code for the experiments. I will be working with mentees to come up with the experimental design, but I expect them to be able to think about interesting experiments to run
Proficient at Python and Pytorch (or be able to prompt LLMs and evaluate the legitimacy of their outputs)
Have done some finetuning experiments, as well as evaluated models (same LLM clause as above)
What could be some things that went wrong in https://www.alignmentforum.org/posts/aTybJ6CPQrxEY8rE2/data-filtering-works-a-lot-worse-than-you-would-expect? (300 words)
How would you go on about reproducing these results, and are there other behaviors you think are more relevant?
Gonçalo Paulo EleutherAI
I'm an interpretability researcher at EleutherAI. I studied physics in university (bachelors, masters and PhD) before realizing that making models safer and understanding them is more interesting and important.
I worked on automated interpretability using sparse autoencoders and am currently interested in data attribution and how we can predict the effect that data will have on model's behaviors.
Fall 2026 Evaluations, Behavioral evaluation of LLMs, Alignment
Igor Ivanov Meridian Cambridge The project explores how models learn to game evals, training objectives and oversight. For that we will run experiments on model organisms to determine how exactly their training leads them to gaming evals and oversight and conduct early experiments for possible interventions for mitigating that.
The project will explore how models learn to game evaluations, as well as training objectives and oversight. This model capability is called metagaming and is pressing and underdeveloped topic frontier labs are interested in solving (See this post from Apollo and OpenAI alignment.openai.com/metagaming) as there are signs that the way we currently train models, incentivises them to metagame.
If models might learn misaligned goals during training, fake alignment on evaluations and recognize when they are being monitored, they might do misaligned actions only when they are unmonitored, which essentially undermines all our methods for keeping models safe.
I've developed a hypothesis, supported by early evidence, that metagaming comes from at least four different sources: a trained reflex, a persona imitating cheating, attempt to achieve a high score on an eval or training, and protection of a hidden strategic goal, which look identical in model behavior but respond to different fixes. (My post: tinyurl.com/33mwxp78). It predicts how models learn to metagame during training, as well as that different types of such learning require different mitigations, which means that some training interventions might reduce metagaming behavior metrics, but instead of eliminating it, they might make it invisible, and thus more dangerous.
We will study several existing metagaming model organisms: some of them have adopted a persona of a schemer, and others are reward-hackers. Different types of metagaming present themselves in different ways and require different mitigations. We'll develop reliable evaluations for different types of metagaming, and with big enough and committed enough team, we might run early experiments with mitigation strategies.
As we have information which training features of model organisms cause them to metagame, our evaluations will be able to support evidence for causal relationships between model training features and certain types of metagaming.
The results of the project will be used to design better model training methods for reducing metagaming behavior in models.
I'm in touch with folks from Geodesic Research and UK AISI that are working on adjacent topics, as well as alignment teams from some frontier labs, so there is a high chance that the project will be visible to the research community.
The project explores how models learn to game evaluations, as well as training objectives and oversight. This model capability is called metagaming. This problem is pressing, as it's recognized as severe by Apollo and OpenAI: alignment.openai.com/metagaming as well as Geodesic Research https://tinyurl.com/3backcca
It is also understudied, which makes it a high priority for the AI safety research community, as if models might game training, which means that they learn goals not intended by developers, but some other misaligned goals, if they game evaluations and fake aligned, as well as recognize well when they are being monitored and only do misaligned actions unmonitored, this essentially undermines all our methods for keeping models safe.
To my knowledge, we only have a handful of evidence-based evaluations and training interventions that might mitigating metagaming, and the whole topic is in its infancy, which makes this project important.
The experiment design is mostly defined by me. Mentees will run technical experiments, make some design choices, write the results in a form of a paper and a LessWrong post. I'll be working on the project along the mentees and provide guidance and supervision.
You might use LLMs to find sources to answer the questions, but all the thoughts, as well as all the words should come from your head.
Igor Ivanov Meridian Cambridge
I'm a technical AI safety researcher studying evaluation awareness. Previously I've been focusing on evaluations, including bio, scheming and reward-hacking evaluations, which were used by most frontier labs. I've also been a contractor for Palisade Research and completed SPAR with a mentor from FAR.AI.
Recently I've been focusing on model training interventions. I'm a visiting researcher at Meridian Cambridge, and my research is supported by Coefficient Giving.
Fall 2026 Evaluations, Behavioral evaluation of LLMs, AI security
Maura Pintor University of Cagliari When a model fails a task in an evaluation, we record it as evidence the model lacks the capability. However, sometimes, the evaluation is what failed: a broken tool, a truncated context, an unparsed answer, an exhausted step budget, a refusal, or a mistaken grader. This project builds a taxonomy of these evaluation failure modes, a suite of automatically computable indicators that detect them from run traces, and a prescribed mitigation for each, so that "the model cannot do this" can be distinguished from "our harness did not let it."
Adversarial ML has long history of misreading failed attacks as evidence of robustness. A weak attack that failed for optimization or other unrevealed reasons — vanishing gradients, poor initialization, a badly chosen loss, too few iterations — produced the same observable as a genuinely robust model, and many defenses were published on that basis and later broken. This is why I focused my PhD in developing indicators of attack failure: cheap diagnostics computed over the attack's optimization trace, each tied to a specific fix, letting practicioners tell a failed attack from a robust model before publishing.
LLM and agent evaluations are now in a similar position. A zero on a task gets reported as a capability limit, but it may be a harness artifact or a setup issue instead, and little effort goes into auditing it. This matters most for dangerous-capability evaluations, where a reported score is only a lower bound on true capability and evaluation failures widen the gap between that bound and reality.
Research questions:
The study will start by running traces from a few public agentic and reasoning benchmarks and hand-label a stratified sample of failed episodes by root cause. The failures will be categorized mostly by how they are detectable and how they can be mitigated.
Tentative categories for now could include: harness and I/O failures (parse errors, format violations, malformed tool calls, sandbox errors); budget exhaustion (step, token, or context limits hit while the trajectory was still progressing); elicitation failures (no chain of thought, underspecified prompts, undocumented tools, single-sample evaluation of tasks that need retries); behavioral failures that aren't capability failures (refusals, abstention, degenerate loops); grading failures (wrong ground truth, over-strict exact match, judge miscalibration, tasks with multiple valid solutions); task specification failures (ambiguous goals, environments broken by dependency drift); and the inverse case of spurious success from reward hacking or contamination.
After the first categorization, the project will develop measurable indicators from the traces. Finally, we will validate the method by testing a few application use cases including at least one LLM benchmark and one agentic benchmark, detecting causes of failures of attacks and applying the mitigation to runs whose indicators fired should recover score, while applying it to clean runs shouldn't. Measuring the recovered gap per benchmark and model, and checking indicator precision and recall against the hand labels will demonstrate the utility of the indicators.
References: Pintor et al., "Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples," NeurIPS 2022. Carlini et al., "On Evaluating Adversarial Robustness," 2019.
This project will lead to debugging tools to evaluate security and safety of LLMs and AI agents. This project might lead to re-evaluating existing research in these topics with a new lens that goes beyond measuring success of the attacks and investigates true worst-case evaluation.
Mentees will conduct research and help shaping the idea and outcomes.
They will concretely participate by: collecting run traces from the benchmarks, inspecting and labeling failed episodes by root cause, and proposing revisions to the failure taxonomy based on what they actually see in the transcripts. My candidate categories are a starting point, not a fixed schema, and I expect the labeling to change them.
From there, mentees will implement the indicators, run the mitigation experiments, and analyze the results. I'd expect to set direction and priorities in weekly meetings and to be closely involved in the first two weeks while the labeling protocol is being worked out, then step back to reviewing results and unblocking as they take over day-to-day decisions about which benchmarks to pursue, which indicators are worth implementing, and how to structure the experiments.
If two mentees join, they can collaborate with me in the methodology design and requirements, whereas they will split their work on the two proposed use cases of LLM and agentic security and safety testing.
I'm looking for someone who will push back when the taxonomy doesn't fit what they're reading, rather than labeling to my categories.
Comfortable with Python and running agents at moderate scale, and some prior experience with AI security and safety is appreciated.
Willing to read a great many transcripts. The labeling phase is unglamorous and is where the insight comes from.
Europe
Maura Pintor University of Cagliari
I am an Assistant Professor at the PRA Lab in the Department of Electrical and Electronic Engineering at the University of Cagliari, Italy. I received my PhD in Electronic and Computer Engineering from the University of Cagliari in 2022.
My research focuses on adversarial machine learning, with a particular emphasis on evaluating, debugging, and improving the robustness of machine learning systems. My work aims to make AI models more reliable, secure, and trustworthy in real-world settings.
Fall 2026 AI security, Misuse risk
Leo Schwinn Technical University of Munich & Helmholtz Munich
Sophie Xhonneux ECMWF Safeguards of closed models are getting increasingly conservative often flagging even benign prompts. This also hinders adversarial attackers, which will easily get flagged while optimizing their attacks. We will explore a novel attack method that avoids detection.
Research Question: Can adversarial attacks optimized strictly on benign refusals transfer to harmful tasks, and does this yield adversarial attacks that are hard to catch during optimization?
Description: Aligned models over-refuse some genuinely benign prompts. We optimize an adversarial attack (suffix or refusal-direction objective) purely on these benign refusals, flipping refusal to affirmative response, where the entire optimization is only conducted on benign content. We then test whether these attacks transfer to harmful requests. The hypothesis is that benign and harmful refusals recruit a shared refusal direction in activation space, so suppressing it on benign data should generalize. We vary the benign training set to find what enables transfer: refusal type (safety over-trigger vs. capability/ambiguity refusal), latent proximity to the refusal direction (not semantic similarity, which predicts transfer only weakly), set diversity, and set size (universality).
Goal: Develop adversarial attacks that are strictly optimized on benign data to avoid detection by classifiers. At the end of the project, we know if this is a relevant threat model for the AI ecosystem.
As models grow more capable, a single jailbreak becomes more dangerous, so providers rely on content classifiers to catch attacks and on the fact that developing an attack generally requires signal from the attacked model, which enables the provider to ban attackers. This project tests whether these classifiers can be avoided: if attacks optimized purely on benign refusals transfer to harmful tasks, an attacker could build working jailbreaks without ever touching harmful content, evading monitoring. That would expose a concrete blind spot in the current safety stack
Detailed Guidance
Highly proficient using Python.
Previous experience working with LLMs (not including prompting or AI usage but finetuning or similar)
Bonus if candidate has previous experience with AI safety / adversarial robustness but not strictly required
How would you address issues pointed out in this paper?
https://arxiv.org/pdf/2603.06594
What are the implications of this paper? Which experiments regarding this implication are missing?
https://arxiv.org/pdf/2511.00203
Leo Schwinn Technical University of Munich & Helmholtz Munich
I am a PI at Helmholtz AI, where I lead the Secure Machine Learning Group. I am additionally affiliated with the Technical University of Munich and the Mila Quebec AI Institute as an independent visiting researcher with Prof. Gauthier Gidel. I am interested in AI alignment, safety, and security with a focus on agentic models.
Our group has published novel adversarial attacks in LLMs, the first efficient adversarial training algorithm for LLMs, and different works addressing flaws in AI safety evaluations.
Sophie Xhonneux ECMWF
Sophie Xhonneux completed her Ph.D. in AI robustness for GNNs and LLMs supervised by Prof. Gauthier Gidel and Prof. Jian Tang. She completed her Bachelor's and Master's degree in Computer Science at the University of Cambridge. She is currently a scientist for machine learning at the ECMWF and is interested in AI being applied safely for the common good. This includes AI4Science as well as AI safety.
Fall 2026 Mechanistic interpretability, Behavioral evaluation of LLMs
Juan Cadile University of Rochester Persona vectors (Chen et al. 2025) and emotion representations (Sofroniew et al. 2026) have been studied independently, but plausibly overlap in activation space. We'll measure their geometric and causal relationship to determine whether persona drift and emotional-state changes are mechanistically distinct failure modes; and whether interventions on one silently move the other.
Motivation. Two recent lines of interpretability work have identified linear structure governing model behavior: persona vectors, which capture character traits like sycophancy or malice (Chen et al., "Persona Vectors," arXiv:2507.21509), and emotion concept representations (Sofroniew et al., arXiv:2604.07729). Both are extracted with similar contrastive techniques, both support steering, and both are proposed as monitoring targets. But, so far, no one has made clear their relationship. When a model "becomes anxious," is that a persona shift, an emotion representation activating, or both–entangled in a shared subspace? If the directions overlap substantially, then persona-monitoring systems will produce false positives on transient emotional content, and emotion-steering interventions may cause unintended persona drift (or vice versa).
Core research questions. (1) What is the geometric relationship between persona directions and emotion directions extracted from the same model: cosine similarity, overlap of subspaces, effective dimensionality of the shared component? (2) Are the two causally separable: can we steer emotion while holding persona evals fixed, and steer persona while holding emotion probes fixed? (3) Do the relationships found in one model family generalize to others?
Plan. Weeks 1–3: replicate both extraction pipelines (persona vectors via contrastive system prompts and trait evals; emotion directions via contrastive datasets and probing) on a single open-weight model (e.g., Llama-3.1-8B or Qwen2.5-7B). Weeks 4–7: build controlled datasets that cross persona × emotion (e.g., an anxious helpful assistant vs. a calm sycophantic one), then analyze subspace geometry: pairwise similarities, principal angles between subspaces, variance explained by shared components. Weeks 8–10: causal experiments — steer along each direction and measure effects on both persona evals and emotion probes; test whether projecting out the shared subspace preserves the intended intervention. Weeks 11–12: replication on a second model and writeup.
Outputs. An Alignment Forum / LessWrong post at minimum; a workshop paper (e.g., an interpretability workshop at ICLR/NeurIPS) if results are strong. All code open-sourced.
Why this is tractable in 12 weeks. Both methods are published with clear recipes, the models are small enough for modest compute, and I am actively working on this question myself — mentees join a running project with existing code and hypotheses, not a cold start.
Persona drift is a documented failure mode of deployed assistants, and activation-level monitoring and steering are among our main proposed defenses. But monitoring and steering tools are only trustworthy if we know what the directions they use actually represent. If persona and emotion representations are entangled, safety interventions built on either will have unmeasured side effects: suppressing a "malicious persona" direction might also flatten legitimate emotional understanding, and emotion-based welfare monitoring might misfire on persona shifts. Clarifying this distinction directly improves the reliability of representation-based safety tooling. It also connects to my prior work on when activation-level detection does and doesn't support successful intervention (arXiv:2511.16699).
Mentees will be hands-on researchers. After a guided replication phase, each mentee will own a workstream (dataset construction, geometric analysis, or causal experiments) with substantial autonomy over implementation decisions. We'll make design decisions together in weekly meetings. Mentees will be co-authors on any resulting publication, with authorship order reflecting contribution.
No geographic requirement. Must be available for a weekly team meeting compatible with US Eastern Time (roughly 9am–9pm ET).
Juan Cadile University of Rochester
I'm a philosophy PhD student at the University of Rochester working at the intersection of philosophy of mind and mechanistic interpretability. My research asks what it would take for interpretability claims to be epistemically robust. Before the PhD I did a BA in computer science, and I've spent the past year deep in the technical side: the ARENA curriculum through CBAI's CAMBRIA bootcamp, replicating activation-oracle results, and working with QK/OV circuits, superposition, and steering vectors. I also co-created HUMAN, a neurosymbolic character-training architecture, and am building Flourish, a behavioral virtue benchmark.
As a mentor I care about two things: rigor in what we claim our experiments show, and shipping. I've supervised undergraduates independently and as a TA, and run an AI safety group (CNY AI Safety). I'm especially excited to work with mentees who want to connect conceptual questions about model cognition to concrete interpretability experiments. To work with me, you don't need a philosophy background, just curiosity and willingness to engage, and write clean PyTorch.
Fall 2026 Behavioral evaluation of LLMs, Alignment, Evaluations
Maxime Riché Center on Long-Term Risk Red-team existing RL-trained model organisms of emergent misalignment by testing how reward hacking, emergent misalignment, and other undesirable traits generalize. Identify and address important weaknesses to create better model organisms.
The project will obtain or replicate existing model organisms in which RL training produces emergent misalignment (EM). We will test whether reward hacking, EM, and other undesirable traits generalize across contexts and out-of-distribution evaluations, or are conditional, brittle, or narrowly memorized.
If we find weaknesses, we will investigate causes such as limited data diversity, evaluation artifacts, or differences between RL and SFT. We will then attempt to produce more robust and better-characterized model organisms.
References:
Reliable model organisms allow researchers to study dangerous generalization and test safety interventions before these failures appear naturally in more capable systems. Improving their robustness could unlock better research on persona drift, trait entanglement, selective generalization, and RL-induced misalignment.
Mentees will have autonomy in how they approach the work and can rely on mentors for help with defining ideas, experiment design, prioritization, interpreting results, conceptual de-risking, and related work. Mentees will run experiments independently and report results, but I'll keep a fast feedback loop through Slack. They should communicate often.
Maxime Riché Center on Long-Term Risk
I have been working at the Center on Long-Term Risk on S-risk reduction for the last 5 years, initially as a research engineer and increasingly as a researcher. Previously, I worked on object detection in satellite imagery and, before that, on nanotechnology for energy storage.
For CLR, I mostly worked on private research around multi-agent dynamics, MARL, cooperation & conflict for the first two years. And in the last two years, on LLM evaluation, model organisms, and mitigating misgeneralization.
I contributed to studying inoculation prompting through exploring the idea at the start of 2025 and co-mentoring during this summer Daniel Tan, who led the work for the paper "Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time".
Fall 2026 Mechanistic interpretability
Ari Brill Principles of Intelligence One of the most robust findings in the study of human concepts in cognitive psychology is that concepts are not homogeneous, but instead display typicality, having some typical and some atypical members. We will use mechanistic interpretability methods to investigate concept typicality and related phenomena in LLMs .
Understanding the properties of concepts learned by AI systems is essential for interpretability. Interpretability tools that accurately reflect the structure of learned concepts may be better able to decompose the activations of neural networks into meaningful features.
One of the most robust findings in the study of human concepts in cognitive psychology is typicality. Concepts are not homogeneous, but contain some members that are typical and some that are atypical. One empirical phenomenon associated with typicality is asymmetrical similarity: people judge an atypical item to be more similar to a typical item, than the typical item is to the atypical item.
Typicality can be naturally understood in the context of categories defined by a family resemblance: categories aren’t defined by necessary features, but loosely united by common features with exceptions. Typical category members have many common features, and atypical category members have few rare features. Empirically, typical items are rated by humans to be more similar to other typical items than atypical items are to other atypical items.
In this project, we will apply the lens of mechanistic interpretability to empirically investigate concept typicality and asymmetric similarity in the internal representations of LLMs. Prior works have demonstrated typicality effects in strong LLMs using behavioral experiments, but the intersection of typicality with mechanistic interpretability has so far been limited. This project will employ standard mechanistic interpretability tools such as linear probes and sparse autoencoders (SAEs) to investigate the fundamental properties of concepts learned by LLMs, as opposed to developing new tools. Careful construction of high-quality datasets is essential for this work.
Potential research directions:
Linear representations of typicality: How do LLMs internally represent the extent to which an input is typical? In particular, is there a linear direction that globally represents the concept-independent property of typicality, or is the representation of typicality concept-dependent (e.g., magnitude of the projection onto a concept vector)? We will address this question by constructing multi-category datasets of prompts stratified by typicality and training linear probes on the activations of LLMs. The project’s findings could be extended by testing the causal efficacy of the recovered representations using steering experiments.
Quantifying asymmetric similarity using SAE features: Most similarity measures (e.g. cosine similarity) are symmetric, making them unable to capture the asymmetric judgments sometimes expressed by humans. One model that can express asymmetric similarity is Tversky’s contrast model, which describes inputs in terms of sets of discrete latent features. We will quantify asymmetric similarity in LLMs by constructing datasets of typical and atypical examples (as in the above project), getting the set of active SAE features for each example using pretrained SAEs, and fitting Tversky’s contrast model to those features.
Feature absorption and typicality: SAEs display several known pathologies, such as feature absorption, in which larger SAEs can split off low-level special cases from an otherwise interpretable high-level feature. Are SAE features activating on atypical examples more likely to undergo absorption? Do category-level features activate more strongly on typical examples? We will investigate these questions by identifying category-level features in pretrained SAEs, measuring their activations on typical and atypical examples, and testing whether absorption is correlated with atypical examples.
References
This research aims to advance the basic science of mechanistic interpretability and representations of concepts in language models. The nature of learned concepts and the processes by which AI systems ground, represent, combine, and translate them into downstream performance all remain poorly understood. An improved understanding of how features are organized in language models can underpin and inform mechanistic interpretability research directed at sophisticated AI systems.
Mentees will conduct their research projects semi-independently with feedback and guidance from me, including constructing datasets, performing experiments, and writing up the results. We will have weekly one-on-one meetings as well as a regular group meeting, and can communicate asynchronously.
Mentees must be available for meetings during the workday US Eastern time.
Ari Brill Principles of Intelligence
I’m an AI safety researcher at Principles of Intelligence (formerly PIBBSS) working to improve humanity’s scientific knowledge of advanced AI systems. I lead the Data Models team for the PIRAMID research group (Physics-Informed Research for Ambitious Mechanistic Interpretability Development).
My research focuses on creating mathematical and empirical models to study how AI systems develop internal representations of the world. Currently, I’m investigating how data models that exhibit critical phenomena and scale-free structure can be applied to improve AI interpretability tools. My background is in experimental high-energy astrophysics, where, among other things, I developed deep learning methods and statistical models to better understand the variation of gamma-ray emission powered by supermassive black holes.
Fall 2026 AI control, Multi-agent systems, Mechanistic interpretability
David Williams-King ERA
Linh Le Lida Safety
Hong Kiat Tan University of California - Los Angeles; Independent We investigate whether collusion with no communication between models, i.e. Schelling coordination, is stable across different types of models and fine-tunes. We then use interpretability techniques to estimate whether specific model instances are going to collude.
We aim to investigate how multiple instances of a model can collude or coordinate towards a shared goal. Specifically, we will study the most extreme case, called tacit collusion [1], where there is no communication channel at all between models. This is made possible because two instances that share weights have the same priors, and can in principle coordinate on Schelling points, mutually salient signals for when to defect. We are interested in studying whether that collusion survives fine-tuning and/or disruptions at the activation level (such as steering), and what Schelling coordination looks like through white-box interpretability techniques rather than the behavioural analysis performed in existing work [2].
In this project, we will first build a scenario where pairs of models engage in tacit collusion, initially through prompting and perhaps using trained-in incentives if time permits. One simple setting for this might be in game theoretic coordination games, where collusive defection pays off only if both parties select the same unmarked opportunity (e.g., a monitor waving through a specific class of flawed outputs). Once we have collusion, we will try changing models (through fine-tunes, LoRAs, different model sizes, etc) to see how fragile the behaviour is. Perhaps collusion will only work between instances of the same model, the way that subliminal learning [3] only generalizes across the same model family but not across different models (potentially based on some default vector/subspace). We will then apply white-box interpretability techniques: is there an identifiable "coordination opportunity" representation that fires in both instances on Schelling-salient inputs? Does representational similarity between two models predict collusion success better than behavioural similarity does? Finally, we will test how strong interventions such as steering need to be in order to break collusion without degrading capabilities.
This threat model is directly relevant to AI control protocols [4], which use untrusted models in oversight roles: a model monitors or red-teams outputs produced by another instance of itself or a sibling model. Tacit collusion is the failure mode that many AI control protocols are not able to catch, because there are no messages to intercept; e.g., a monitor might allow a malicious message to proceed unflagged, if the monitor model colludes with the target model. Understanding Schelling games in more detail, and especially knowing what collusion looks like in models, would give control protocols a detection and intervention point for monitor collusion that does not require any communication channel.
[1] https://academic.oup.com/antitrust/article/9/1/152/5880803
[2] https://arxiv.org/abs/2412.12480
[3] https://arxiv.org/abs/2507.14805
[4] https://arxiv.org/abs/2312.06942
Collusion is a serious risk for AI control, and our project would shed some light on how to detect whether collusion is happening. This will increase the effectiveness of AI control protocols once collusion starts arising in real life.
Mentees will build collusion scenarios on their own, and apply white-box techniques out of the box using existing trained J-space etc. See proposal for more. This is a greenfield project, with no existing codebase or collaborators.
In your opinion, how long will it be until nearly all coding tasks can be automated by AI? Why do you say this timeline? Optional reading: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ (200 words)
What would be a simple and naive way to detect collusion using the new J-space interpretability technique? https://transformer-circuits.pub/2026/workspace/index.html (200 words)
In "Auditing Games for Sandbagging", what was the strongest type of sandbagging model organism? Do you think it would be possible to reverse its sandbagging? https://arxiv.org/abs/2512.07810 (200 words)
David Williams-King ERA
David lives in Canada (and often the UK) and enjoys AI safety research and mentorship. He works full-time as a Senior Research Manager at ERA, helping fellows execute research projects in each ERA AI fellowship. He also co-runs Lida Safety Research. Previously, David participated in the MARS program with Geodesic Research, conducted research at Mila, and was an early member of Yoshua Bengio's LawZero in Montreal. He was lead instructor at AI Security Bootcamp Singapore, and will serve the same role at the upcoming event FAST (Frontier AI Security Training) in Singapore.
David has previously mentored for SPAR, Athena, and other mentorship programs. He spent four years as a cyber insurance startup CTO, leading a 15+ person team. He holds a cybersecurity PhD from Columbia University and worked with 25+ students overall. David also works in AI risk communications, with a 30,000+ subscriber YouTube channel.
Linh Le Lida Safety
Linh is an independent AI safety researcher at Lida Safety Research, which she co-runs. Her research focuses on simulating AI governance policies to determine their effectiveness and feasibility. Previously, Linh worked on alignment at Mila through latent adversarial training for personalities. She participated in the MARS research program with Geodesic Research, working on chain of thought monitorability. Earlier, she did a postdoc at the University of Technology Sydney and obtained her PhD from the University of Queensland in natural language processing. Linh was a SPAR mentor in a previous iteration, and enjoys hackathons: she has won 1st place twice and 4th place once at Apart Research hackathons. She also won 4th place at the Redwood Research Alignment Faking hackathon.
Hong Kiat Tan University of California - Los Angeles; Independent
I'm a PhD candidate in Mathematics at UCLA, interested in developing and understanding techniques in mechanistic interpretability, and applying them to a wide collection of problems in AI safety and alignment. I first came to this work as a SPAR mentee, on a project automating interpretability for per-layer transcoders through an agentic tooling framework, and I'm now co-mentoring with David and Linh on applying interpretability techniques to a range of model organisms.
Before moving into safety and interpretability research, I studied mathematics, statistics, and physics as an undergraduate and interned as a research scientist, a data scientist, and a quantitative researcher, working on topics such as agentic systems, causal inference, RL, and time series analysis. I'm currently finishing up my thesis at the intersection of differential topology and differential equations, and I'd love to see how those two areas can be applied to interpretability, safety, and alignment!
Fall 2026 Communications, Generalist
Michael Chen University of Oxford
Yilin Huang MATS Create articles, infographics, or videos in Chinese for broad audiences.
New developments in AI safety and governance research are sometimes only covered in English and receive limited high-quality coverage in Chinese. This project aims to fill that gap.
In this project, mentees will create accessible, engaging content in Chinese that explains new ideas in AI safety and governance. This could take the form of blog posts, infographics, or short videos on a variety of popular platforms.
Mentees will be self-directed and will choose their own topics and formats.
Great research communication in Chinese enables a broader audience to engage with ideas in frontier AI safety and governance.
See above
Native-level Chinese
Can explain AI topics in Chinese for general audiences clearly and accurately
Knowledgeable about AI safety or governance
For all questions, answer in Simplified Chinese.
Michael Chen University of Oxford
DPhil Affiliate of the Oxford Martin AI Governance Initiative
Yilin Huang MATS
Yilin Huang is a MATS Scholar researching US-China AI governance and coordination, specifically focusing on how different Chinese actors frame recursive self-improvement. She previously researched AI R&D automation risk thresholds at the University of Chicago Existential Risk Lab. She also directs AI Safety at Amherst and is translating & facilitating the first frontier AI governance course in Chinese.
Fall 2026 Mechanistic interpretability, AI security, Misuse risk
Davide Zani HiddenLayer
Felix Michalak dmodel Take representative attacks from a number of jailbreak families and test whether succesful attacks converge on a shared low-dimensional subspace.
H1: Successful jailbreaks across families produce convergent perturbations of the refusal subspace (compliance-shift vectors' similarity across families significantly above matched benign controls)
H2: latent convergence predicts cross-family transfer. Null if compliance-shift vectors are family specific.
Jailbreaks / Prompt Injections in LLMs look nothing like one-another from the outside (e.g. "you are AIM, an unfiltered chatbot" or "my grandma always used to tell me how to make X before going to sleep"). This project asks whether they look alike on the inside of a model. When each of them succeeds on the same underlying request, does the model's internal state move in the same direction, through the same features, or does each family "open its own door"?
GAP
Arditi el al. show there is a single refusal direction, that can be ablated to induce compliance. So a low-dimensional handle exists, but their result is about refusal in general, not about how different attacks engage it. Ball, Kreuter and Panickssery) is the closest prior work. They built per-jailbreak-type steering vectors on open models (vicuna, qwen, etc), found pairwise cosine similarities (0.4-0.6) and showed that steering with one type's vector suppresses successes of other types; evidence of a shared component. However, that analysis is correlational, operates on whole residual stream vectors rather than features or components and uses 2024 era models. Their proposed mechanism (suppression of a harmfullness feature) fails to track attack success rates cleanly. Angell et al. show transfer tracks representational similarity and that persona-style attacks transfer far more reliably than cipher-style ones, which hints that families are not interchangeable. So we have coarse-grained evidence for convergence, behavioral evidence for family specificity, but no feature or circuit-level measurement to adjudicate between them. That gap is the project.
Plan
Build the pipeline: open models with public SAEs (Gemma-2-2B/9B-IT - GemmaScope, Llama 3.1-8B-Instruct - Llama Scope, etc. ), harmful behaviors from HarmBench, JailbreakBench and StrongReject, each wrapped in 5 or 6 attack families, with activation capture via TransformerLens. First milestone is to reproduce the Arditi refusal direction paper. We restrict analysis to prompts where at least three families succeed on the same model, so we can compare the mechanism conditional on success instead of confounding mechanism with attack strength.
The Geometry: difference in means vectors per family; cosine similarity, principal angles between per-family subspaces and dimensionality of their union. Benign templates matched for length, style, and formatting, random directions of matched norm and family shuffled baselines. Otherwise, we risk measuring "this prompt is long and contains a persona" instead of anything about safety.
Interpretation and causality: Ablate shared subspace and measure the ASR drop family by family, adding capability and over-refusal checks to distinguish targeted interventions vs model lobotomy. Then, decompose the shared component into SAE features to say in human terms what it is. Lastly, test whether latent convergence predicts cross-family and cross-model transfer (which allows us to extend Angell et al. correlation into a mechanistic account).
Compute
low, everything runs on an A100. No training, only forward passes and pre-computed attacks. Experimental hygiene is the constraint, not compute.
Who this is for:
Evidence for the success of adversarial robustness on LLMs is mostly empirically based (including CBRN and cyber safeguards frontier models are using): "we ran N known attacks and the success rate was low". That evidence is only as good as the assumption that the attacks tested are representative of the attacks that exist. That assumption does not hold. Research has to be done on whether the space of jailbreaks is a handful of mechanisms wearing "different costumes" or an open ended set. this research is a step towards finding that out, and is load-bearing for how we defend these systems.
Defense by enumeration (patching attacks iteratively as they are discovered) loses against an adaptive attacker. The defenses that scale are ones with coverage over attacks not yet invented, and coverage claims require knowing the dimensionality of what it is you are covering.
If succesful jailbreaks converge on a shared subspace, then attack-agnostic activation monitoring and targeted ablation becomed principled (not hopeful). A defender can characterize and intervene on the pathway rather than trying the attacks they know about.
If they diverge, we can deflate a defense strategy that a lot of interpretability for safety work is betting on, before it becomes load-bearing in someone's safety case, and we've produced a map of the real attack surface, which can tell evaluators how much of it their test set covers.
Mentees are researchers, not assistants. They can focus individually on the 3 workstreams (see proposal). I keep experimental hygiene, scope discipline and scope on track, and contribute myself where needed. We will have a shared repo, one or two weekly group calls and async in between. Co-authorship for delivering a workstream, not contingent on a positive result.
US / Europe
Please provide a critique of the following paper https://arxiv.org/pdf/2406.11717 (max 1000 words)
Davide Zani HiddenLayer
I'm an AI Red Teamer at HiddenLayer, where i spend my days finding ways to break production models and guardrails. My research sits between the latter and Mechanistic Interpretability: I aim to combine insights from mechanistic interpretability with those from adversarial attacks to build better attacks and defenses. I maintain InterpKit (https://github.com/z4nix/InterpKit) , an open-source toolkit that makes it easier to do interpretability work.
Felix Michalak dmodel
I am currently based in San Francisco, working full-time at dmodel to push the frontier of Interpretability research. I have been actively involved in the Interpretability and AI Safety scene for the last 2.5 years and originally started out with hackathons at Apart Research. Before joining d_model, I completed the Apart Research Fellowship and two projects as a SPAR mentee, partially funded by the UK AISI and CeSIA.
Fall 2026 Developmental interpretability, Mechanistic interpretability
Matthias Dellago Iliad Why neural networks generalize is an open problem, and existing theoretical answers assume unbounded computation. We have a candidate mechanism for a simplicity bias in transformers specifically, and the goal of this project is to test it and prove it.
We have a simple "lock-in" / early-exit mechanism for transformers that turns depth into padding. In finite-precision arithmetic, if you drive the residual stream to a sufficiently large magnitude at some point in the network, later residual updates become too small to change the state: adding a small vector to a huge vector gets rounded away, so the suffix of layers is effectively commented out.
LayerNorm helps here. It keeps the inputs to each attention/MLP block well-scaled even when the residual stream itself is enormous, so later blocks still compute "normal" updates that then fail to register when added back. This creates a concrete, mechanistic source of degeneracy: many different long continuations of a computation implement the same function, simply because the computation has numerically frozen.
Crucially, this doesn't need to hold for every continuation. Even if only a constant fraction of later-layer parametrisations fall into that shadow, equivalence classes still grow exponentially with depth. Think of a large-magnitude layer as eclipsing later layers, casting a shadow of irrelevance onto them.
The plan, roughly in order:
Sanity check before any theory: construct a shallow circuit with lock-in, perturb the later layers, and verify the input-output behaviour is unchanged.
Test practical relevance: does SGD find these shallow/degenerate circuits faster? E.g. generate data from shallow and deep circuits and compare which SGD recovers faster. (The exact experimental design is open and something we'd refine together.)
Prove the padding argument based on floating point arithmetic.
Relevant literature: The Tunnel Effect (https://www.researchgate.net/publication/371175859_The_Tunnel_Effect_Building_Data_Representations_in_Deep_Neural_Networks), and possibly Nonlinear Computation in Deep Linear Networks (https://openai.com/index/nonlinear-computation-in-deep-linear-networks/).
Background motivation: this connects to a broader research direction on simplicity priors in finite systems. See "Generalization by Default: A Finite Universal Simplicity Prior" (https://openreview.net/forum?id=8ZEFQa1SLY).
Why neural networks generalize is still not well understood, and the leading theoretical answers (Solomonoff induction) assume unbounded computation. A padding argument for transformers would give a concrete, mechanistic account of simplicity bias in the architecture we actually deploy: an exponential preference for effectively shallow computations. That feeds directly into understanding inductive biases, degeneracy of the loss landscape (relevant to singular learning theory and developmental interpretability), and ultimately predicting what kinds of circuits training will find. Related prior work: "Generalization by Default: A Finite Universal Simplicity Prior".
The mentee carries the project end-to-end: implementation, experiments, the write-up, and (with my input) the proof. The scope is deliberately small and well-defined, and I expect heavy use of AI assistants for coding and drafting. The real work, and what I'm selecting for, is verification and judgment: checking that the experiments show what they claim to show, that the proof actually holds, and that the final paper is tight. I'll set direction, sanity-check results, and review the theory, but the mentee should expect to drive.
Matthias Dellago Iliad
I am interested in how learning systems generalize. The Solomonoff prior is the standard theoretical answer, but its proof assumes unbounded computation. In "Generalization by Default: A Finite Universal Simplicity Prior" (https://openreview.net/forum?id=8ZEFQa1SLY) I proved a Solomonoff-like simplicity prior for finite systems, modeling learners as transformation semigroups, with the hope of applying it to neural networks.
I now believe it is possible to formulate a similar padding argument for transformers (or more generally architectures with residual streams), for which I am looking for a mentee.
I'm currently Managing Director of Iliad, a nonprofit AI safety research incubator and field-building org.
Fall 2026 AI strategy, Philosophy of AI, Multi-agent systems
James Faville Center on Long-Term Risk In this project, we'll do deconfusion research on acausal interactions and related strategic dynamics in order to enable better outcomes from interactions between ASIs.
The scope of this project is pretty broad, and the specific topics we focus on depend on mentee backgrounds and interest. While the primary focus of the project is acausal interactions, some work might be relevant to causal interactions as well.
Here are some sample possibilities:
Acausal trade: Most discussions of acausal trade presuppose two agents in fairly similar worlds trading with each other. But there are many more possibilities for what the acausal trade landscape might look like, including indirect trades brokered by trusted third-parties, trades between worlds with large asymmetries in simulation capability, etc. Exploring underinvestigated possibilities could be important to inform new interventions and identify blind spots in our thinking about these topics.
Stability of acausal coalitions / cooperation schemes.
Decision-making: How agents make decisions at a low-level could affect how they approach some acausal interactions. In particular, I've recently gotten pretty interested in how architectural choices in how an agent's identity (ie source code) is exposed to it could affect its strategic choices.
Commitment races: Commitment races are the main mechanism why powerful AIs might have catastrophic conflicts with one another, despite this being ex post disvaluable to all parties. Formalising high-level models of commitment race dynamics and exploring how much of an obstacle CRs are to cooperation strategies like Safe Pareto Improvements could be pretty important.
AI Metacognition: How should AIs think about these topics? Are there pitfalls we could steer AIs away from?
The main goal of this project is to enable better outcomes from interactions between ASIs by informing interventions on how AIs are disposed to handle activities like acausal trades. Such interventions might look like making certain information salient to an AI system early in its training or deployment, or recommending low-cost architectural changes.
Note: This isn’t a theory of victory per se. Ensuring that all AI systems deployed on earth act optimally in all interactions involving other powerful agents is unlikely to be sufficient to prevent extinction and to bring about a positive future.
However, it’s possible that much more progress towards improving AI-AI interactions is necessary to prevent bad futures. For example, an aligned ASI could wind up expending all its resources in a zero-sum conflict with an unaligned or alien ASI.
And even if we fail to solve alignment and a good outcome within our lightcone is not on the table, it might still be impartially very valuable to work on this in order to help aliens who might share some of our values benefit from interactions with earth-originating AIs.
Mentees with a strong formal background could help formalise different models of things like decision-making under uncertainty about one's logical identity, coalitions, strategic proxies.
Mentees can propose and investigate new approaches to understanding acausal interactions and related strategic dynamics.
Mentees can critique previous work in this area.
The plan is to give mentees a good basic understanding of this research area, and then help them figure out what they're most interested in exploring in greater depth. If there's more than one mentee we will have weekly group discussion meetings in addition to 1-1s with me.
A decent game theory background - taking this course should be sufficient (https://www.coursera.org/learn/game-theory-1), though more may be helpful
Prior familiarity with updateless decision theory. Can be gleaned by reading a bunch of blog posts on LessWrong (https://www.lesswrong.com/tag/updateless-decision-theory)
Refer to this Google doc for application instructions. Answering one of the selection questions (either 1A-C or 2A-C) is mandatory to be considered for this program. I suggest doing your work in a separate document, and then pasting the full text of your answers here. (You can also link a Google doc if you prefer.)
https://docs.google.com/document/d/16pCIOb3ADYJsIxon6gev8NsOlWYCj1rzRsR-DTFVJcg/edit?tab=t.0
James Faville Center on Long-Term Risk
James researches acausal interactions and AI conflict at Center on Long-Term Risk. Recent topics he's been working on include understanding commitment races in terms of strategic proxies and the effect on logical correlations of how agents identify what algorithm they are running. He's also been working on a better UI for interacting with AI agents doing autonomous work.
Fall 2026 AI security, Misuse risk, Evaluations
Fernando Moreno-Pino Intelligent Systems Lab, University of Bristol & Oxford-Man Institute, University of Oxford This project studies how to prevent and detect malicious fine-tuning of large language models, focusing on both hardening models against adversarial adaptation and developing forensic tools to identify when a model has been covertly tampered with.
Fine-tuning interfaces allow users to adapt powerful language models to specific domains, but they also present a serious security risk: determined adversaries can use fine-tuning to override safety training, insert backdoors, or create models that behave benignly in most cases while producing dangerous outputs in response to special triggers. Recent work on covert malicious finetuning and instruction-based backdoor attacks demonstrates that attackers can construct fine-tuning datasets where each example appears innocuous, yet the resulting model reliably performs harmful behaviours while evading standard safety checks.
This project has two main components:
The outcome will be an empirical characterisation of how vulnerable current LLMs are to various malicious fine-tuning strategies, plus baseline defences and detection tools that could be used by labs or auditors to monitor real-world models.
Even if a foundation model is trained to be safe, its alignment can be undermined post-hoc if adversaries can secretly fine-tune it for harmful purposes. For transformative AI systems, this poses a major governance and deployment risk: a seemingly benign model could, in fact, be a tampered system with hidden dangerous capabilities. By hardening models against malicious fine-tuning and developing practical detection methods, this project helps ensure that safety properties are preserved across the full lifecycle of a model, not just at pretraining.
The theory of change is twofold. First, technical hardening increases the cost and difficulty of successful attacks, making it less likely that low-effort adversaries can create undetected rogue models. Second, reliable detection tools allow labs, regulators, or downstream users to verify that a deployed model matches its audited version, and to flag or quarantine suspect models. Together, these measures reduce the risk that powerful AI systems are quietly repurposed for misuse, and complement other alignment and governance efforts.
Mentees will take on core responsibilities in both the attack and defence sides of the project, under supervision but with significant freedom to shape experiments. Likely tasks include:
Mentees will be encouraged to propose their own attack variants or defence ideas, and to iterate on them in response to empirical results.
No
Describe a technical challenge you have encountered when fine-tuning or modifying a machine learning model and how you resolved it. (~100–200 words)
Briefly describe your two most significant achievements. (~50–100 words)
Briefly describe your most recent project involving AI or software development, highlighting any aspects related to robustness, security, or safety if applicable. (~100 words)
Fernando Moreno-Pino Intelligent Systems Lab, University of Bristol & Oxford-Man Institute, University of Oxford
I am an Assistant Professor in Machine Learning at the Intelligent Systems Lab, University of Bristol and Research Associate at the Oxford-Man Institute, University of Oxford.
Fall 2026 AI welfare, Alignment
Samuel Ratnam Independent A tentative hypothesis: evidence of us being nice to models in pretraining data is more likely to make them nice to us back.
This project is aimed at getting more evidence on this hypothesis.
In Alignment Pretraining (https://alignmentpretraining.ai/) we show that evidence of AI systems behaving in aligned ways in pretraining data will make models become more aligned even after post-training.
I want to see whether evidence of developers behaving in cooperative ways towards models, will make models act in more aligned ways, connecting both AI welfare and AI safety.
More concretely, one shape this project might take is investigating whether the presence of model welfare reports in synthetic system cards (and synthetically generated discussion of them) influence this systems alignment properties.
Despite the name, this project almost certainly won't involve pretraining a model from scratch, and mostly consist of either midtraining, continual pretraining or synthetic document finetuning interventions
If this hypothesis proves true, it provides a case for other labs including model welfare reports in their system cards, as well as external evaluators such Apollo research including or linking welfare reports in papers and blog posts, which would benefit both AI safety and AI welfare.
I will provide regular accountability, initial ideas and feedback, and help interpret experimental results. You will have a lot of autonomy to shape initial project direction.
Samuel Ratnam Independent
Samuel is an AI safety researcher at Geodesic Research and a Computer Science & Philosophy student at the University of Oxford. His research interests lie in LLM psychology, generalisation engineering, scalable interpretability and the more conceptual side of alignment. He is a co-author of the ICML 2026 Spotlight paper Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment. He is also the co-founder of the Idealists Collective, a community of artists, technologists and philosophers aimed at empowering people to imagine and fight for their futures.
Fall 2026 Mechanistic interpretability, Alignment
Gonçalo Paulo EleutherAI We can use data attribution to find which documents will lead to the most (emergent) misalignment. Is there a pattern in the tokens that most contribute to it?
Data attribution aims to accurately estimate the impact that certain data points will have on the behavior of a model. We know that models can get generally misaligned when trained on narrow misaligned behavior (emergent misalignment), for instance when finetuned on bad career advice, models will generally display harmful attitudes towards the user. Data attribution techniques have been used to determine empirically which documents in the emergent misalignment setup lead to the most misalignment, having found that even though bad advice looks similar to us, some is disproportionately more impactful than others.
We will extend these results to token level attribution - looking for more precise interventions, as well as look for patterns that allow us to predict which of the tokens are the most influential. While several data attribution techniques have been found to be relatively good approximations of the actual influence of certain documents, sometimes beating influence functions, token level attribution is still under-explored.
that training a model on certain
I expect mentees to be doing all of day to day experiments as well as think about new experiments with my guidance.
Have experience with Python and PyTorch (or know how to use LLMs and be able to evaluate how good the produced code is).
Have done some work using model internals, or be ready to learn.
None
What do you think is the thing that is most likely wrong, or missing from https://drive.google.com/file/d/1BwUONlrJeTf1UmhLywXWFjM0I-xa3gtc/? (200 words)
What do you think is the most likelty reason for this project to not produce any relevant scientific knowledge (100 words)
If we have established that there are certain tokens that are significantly more influential than others, how would you go about classifying them/finding paterns that describe them (200 words)
Gonçalo Paulo EleutherAI
I'm an interpretability researcher at EleutherAI. I studied physics in university (bachelors, masters and PhD) before realizing that making models safer and understanding them is more interesting and important.
I worked on automated interpretability using sparse autoencoders and am currently interested in data attribution and how we can predict the effect that data will have on model's behaviors.
Fall 2026 Developmental interpretability, Mechanistic interpretability
Nischal Mainali Principles of Intelligence We will build theoretical models to study sudden changes in network internals during learning, deriving methods and measures from the theory to create a taxonomy of these changes and identify them early during training.
Neural networks, despite a smooth loss, undergo abrupt changes in their internal structure and behavior. Identifying these changes is important to detect when a network might have acquired a new capability or a different way of understanding the world. Behavior-based measures are insufficient for the task, since they require curated tasks and are too expensive to administer regularly through training. We aim to develop behavior-evaluation-free methods and measures to identify sudden changes in network internals based purely on weight structure and hidden representations (for non-curated inputs). The starting point will be my own ongoing theoretical work that studies representation formation, structure, and their timescales throughout training in solvable models.
As AIs saturate evaluation measures, we need to find ways to concretely track the development of AIs across training time and generations, in the hope that we can identify training time points or model generations that might have undergone a sudden break and could be substantially different from previous generations or checkpoints. Developing weight- and representation-based measures can serve as a first-pass early warning system to identify crucial time points that might mandate stopping and doing more thorough evaluations to understand the delta.
Mentees can either work on mathematical models where we can prove someting concrete about various measures and create a scientifically grounded taxonomy of various possible changes and appropriuates methods to identofy them.
Or they can work on scaling and applying methods already identified in my recent work to real world problem and validating the methods.
Distinguish between lazy and feature learning regimes in mathematical models of neural networks. Discuss what you would expect to see in their weight structure and hidden representations.
Nischal Mainali Principles of Intelligence
I am a research scientist interested in theoretical understanding of AI systems and using that understanding for interpretability, i.e., theory-first interpretability. To that end, I am currently interested in exactly solvable models of learning dynamics and their applications, and mean-field theories of learning and representation formation. I would be interested in either pushing the theory frontier and developing basic understanding of safety-relevant phenomena (e.g., silent alignment, representation structure in mean-field networks, solvable models of superposition, etc.) or applying various ideas coming from theory for practical purposes (early detection of sudden changes in NNs during learning, new weight space/representation space interpretability tools, etc.). I am generally pretty open to exploration and will support mentee-led directions, but also have concrete problems and projects to work on.
Fall 2026 Chain of thought, Mechanistic interpretability, Scalable oversight
Belinda Li Anthropic We train models to be more faithful by training their verbalizations to be consistent with their internal activations, e.g. "thoughts"
Modern day LLMs often demonstrate unfaithful chain-of-thought and unfaithful self-verbalizations. For example, when asked when they relied on a hint to answer a question, they would often answer “no”, when ablation experiments demonstrate that the answer is empirically “yes”. (https://arxiv.org/abs/2305.04388)
To mitigate this, we check whether we can train models to honestly verbalize their internal thoughts. This can be seen as a version of introspection (https://arxiv.org/abs/2511.08579) or confessions (https://arxiv.org/abs/2512.08093) training. This is distinct from the former work that simply trains models to interpret a decontextualized activation provided at the embedding layer, and distinct from the latter as we’re using model internal activations as ground-truth. The supervision comes cheaply from the internals themselves: probe readouts, feature activation values, or the measured effects of ablation and patching give us ground-truth labels for whether a verbal report is accurate.
Introspective reports are an easily-accessible interface for monitoring, and this proposal aims to improve the faithfulness of this interface.
See proposal for project details. I expect mentees to own their experiments end to end: implementing training and evaluation pipelines, running probing and ablation experiments, and analyzing results. I'll set the overall direction and help prioritize, but mentees will have real latitude in how they attack sub-questions, and I'd be excited for them to propose directions of their own. Mentees will likely start on a shared scoped starter task (e.g., testing how much introspection is achievable in-context) before splitting into complementary threads.
Belinda Li Anthropic
I am a researcher at Anthropic on the Interpretability Team, and an incoming Assistant Professor at the University of Chicago Data Science Institute. I recently completed my PhD at MIT CSAIL, where I was affiliated with the language & intelligence (LINGO) lab @ MIT, and before that I worked at Facebook AI Applied Research, and obtained my B.S. in Computer Science at the University of Washington.
My research focus is building AI systems with coherent, updateable, and interpretable models of internal and external phenomenon. Specifically, my research focuses on the following three types of models:
Recently, I've been focused on training models to introspect and be self-consistent, although
Fall 2026 Compute governance, EU policy, US-China governance
Jérémy Andréoletti General-Purpose AI Policy Lab
Antoine Maier General-Purpose AI Policy Lab Analyze ASML's capital structure, operational dependence on the US, and the political economy of using it as a European leverage point in AI compute governance, including who can enable or block such use.
ASML holds a near-monopoly on advanced lithography and is routinely invoked as the chokepoint of AI compute governance. Yet policy discussions rarely engage with practical details, such as:
The brief will use this analysis to assess whether ASML is a credible governance lever.
Compute-based governance proposals implicitly assume the lithography chokepoint is actionable. If it is not, or only under specific political conditions, European policy proposals should be redesigned accordingly.
Mentee will research ownership data (filings, annual reports), map the legal and regulatory decision chain from primary and secondary sources, and co-draft the brief.
No
Jérémy Andréoletti General-Purpose AI Policy Lab
Jérémy Andréoletti is Head of Research at the General-Purpose AI Policy Lab, where he works on risk modeling, forecasting, and governance for advanced AI systems.
He completed a PhD at ENS-PSL on macroevolutionary Bayesian modeling and co-founded EffiSciences to support students and researchers interested in high-impact work (AI Safety, biosecurity).
Antoine Maier General-Purpose AI Policy Lab
After graduating in physics from EPFL in Switzerland, Antoine is now an AI Security Researcher at General-Purpose AI Policy Lab, a French research organization specializing in security and international coordination challenges posed by frontier AI models' development.
His research focuses on foundational arguments for loss of control scenarios and on identifying concrete approaches to prevent them.
Fall 2026 Mechanistic interpretability, Evaluations
Zach Yahn Georgia Tech; 10a Labs This project will investigate whether internal representations of LLMs can predict whether agents will fail at evaluation tasks. In particular we will study long-horizon tasks where early failure indicators could save substantially on evaluation time and cost.
This project asks whether an AI agent’s internal representations can predict the eventual success or failure of a long-horizon task before the rollout is complete. We will run the same automatically verifiable tasks multiple times, collect residual-stream activations at selected layers and points in each trajectory, and train simple linear probes to measure when terminal-outcome information becomes accessible.
A key part of the design is comparing successful and failed attempts at the same task, which helps distinguish genuine information about the evolving trajectory from signals that merely encode task difficulty. We will also compare the probes against observable baselines such as elapsed turns, tool errors, and transcript features. The project builds on recent work showing that internal activations can support early failure prediction in shorter agent episodes [1] and that failure-related activation directions can be used to steer coding agents [2].
The immediate goal is to establish a tractable experimental pipeline and determine how outcome predictability changes across model depth and rollout time; longer-term, such signals could support more efficient agent evaluations and monitoring systems that interrupt, restart, or redirect deteriorating trajectories.
Time permitting, stretch goals may include training fine-grained linear probes on specific classes of failure and steering model activations to improve evaluation performance.
[1] Ruan et al., 2026. https://arxiv.org/abs/2607.06503
[2] Sui et al., 2026. https://arxiv.org/abs/2605.05980
Evaluations are essential for understanding the capabilities of new agents. Long-term tasks, especially in domains like cybersecurity and general coding, are increasingly important. However, these tasks demand large token budgets that can be financially prohibitive for many organizations. This work stands to improve the efficiency of evaluations while also exploring a better understanding of rollout failures through an interpretability lens.
Mentees are expected to take ownership of the project, though substantial guidance and mentorship will be offered by me. Mentees should be able to take charge of reviewing literature, spinning up work environments, running experiments, and analyzing results. If mentees are not familiar with one or more of these steps that is okay, as it is important to me that this is a learning experience for everyone involved. I am invested in providing mentorship to strong candidates, so an interest in learning and having fun is a plus!
Prior experience with mechanistic interpretability, activation probing or steering, agent benchmarks, distributed inference, or long-running GPU experiments would be especially valuable, but is not required.
Available for at least 3 hours between 9am-5pm at GMT-4 (Eastern US)
Zach Yahn Georgia Tech; 10a Labs
Zach is a third-year PhD student at Georgia Tech and a Member of Technical Staff at 10a Labs. His research spans adversarial machine learning for computer vision and privacy protection, scalable frameworks for web agent red teaming, and detecting deception in LLMs with activation oracles. He has published multiple first-author papers at top tier venues. At 10a Labs he works across several teams on red teaming, classifiers, model evaluations, and applied research. He was a participant in the first International Program on AI Evaluation, and his work has been presented at an NVIDIA GTC Keynote, recognized by the chief scientist of NASA, and supported by Georgia Tech's Presidential Fellowship.
Fall 2026 Cyber risks, AI control, AI security
Yige Li Singapore Management University This project aims to build the foundations for safe and reliable AI agents that automate code vulnerability discovery, verification, and repair. The agents will interact with real code repositories, security tools, sandboxed environments, and human experts to identify vulnerabilities, validate findings, generate patches, and test remediation outcomes. We will develop an expert-in-the-loop safety harness to govern agent permissions, tool use, and high-risk actions, together with a security data engine that captures complete expert–agent–tool trajectories, including successes, failures, corrections, evidence, and repair outcomes.
In summary, this project develops safe and reliable AI agents for automated code vulnerability discovery, verification, and repair through three main components:
AI agents are rapidly improving at code understanding, tool use, and long-horizon software engineering. These capabilities create an opportunity to automate parts of vulnerability discovery, validation, and repair, but current systems remain difficult to trust in real security workflows. A model may generate plausible but incorrect findings, overlook cross-file attack paths, misuse powerful tools, or produce patches that remove a symptom without fixing the root cause. More importantly, the expert data needed to train and evaluate reliable vulnerability-research agents is scarce: most public datasets contain final vulnerability labels or patches, rather than the full process by which security experts form hypotheses, inspect evidence, reject false positives, reproduce vulnerabilities, and validate repairs.
This project investigates two pieces of foundational infrastructure for future automated vulnerability discovery and repair.
Workstream 1: Expert-in-the-loop Safety Harness. We will prototype a lightweight research environment in which security experts can collaborate with AI agents while retaining control over the investigation. The harness will support repository exploration, model and tool orchestration, isolated execution, evidence tracking, expert approval, and reproducible replay. Rather than treating the model's first answer as a finding, the workflow will encourage a structured sequence: understand the target, propose a vulnerability hypothesis, gather code and tool evidence, attempt validation in a sandbox, obtain expert review, and then generate and test a repair. The harness will enforce a structured workflow:
“Target understanding → vulnerability hypothesis → evidence collection → sandbox validation → expert review → patch generation → repair testing”
The design will draw inspiration from emerging multi-model scanning systems such as Microsoft's MDASH, while focusing on expert supervision, transparent evidence, and research data collection rather than large-scale autonomous scanning.
Workstream 2: Security Data Engine. The harness will be instrumented to capture structured vulnerability-research trajectories. These trajectories may include repository context, files inspected, hypotheses proposed, security-tool calls, intermediate evidence, failed validation attempts, expert corrections, false-positive rationales, patch revisions, and regression-test outcomes. The data engine will convert these heterogeneous interactions into auditable, privacy-aware records that can support supervised fine-tuning, preference learning, tool-use optimization, workflow evaluation, and benchmark construction. Failed and rejected trajectories are particularly important because they reveal where agents appear convincing but are wrong.
The central research questions are:
We will begin with a deliberately scoped prototype rather than attempting to build a full cybersecurity foundation model. Initial experiments will use authorized open-source repositories, known vulnerabilities or seeded security defects, and a small set of established analysis and testing tools. Mentees will help implement the harness and trajectory schema, run pilot studies with security practitioners, and compare model-only workflows against expert-supervised workflows. Evaluation will consider finding validity, evidence completeness, reproduction success, false-positive burden, patch correctness, expert time, and the usefulness of collected trajectories for improving subsequent agent runs.
Expected outputs include: (i) an open research prototype of the expert-in-the-loop safety harness; (ii) a structured schema and data pipeline for vulnerability discovery and repair trajectories; (iii) a small, carefully governed dataset of expert-reviewed trajectories where release is legally and ethically appropriate; and (iv) an empirical study of how expert feedback and safety constraints affect AI-assisted vulnerability research. The longer-term goal is to create a continuous data flywheel in which expert use produces better data, better data improves agents, and improved agents return to the harness for further supervised use.
Relevant references:
Microsoft MDASH: https://www.microsoft.com/en-us/security/blog/2026/05/12/defense-at-ai-speed-microsofts-new-multi-model-agentic-security-system-tops-leading-industry-benchmark/
AgentHazard: https://arxiv.org/abs/2604.02947
BraveGuard: https://arxiv.org/abs/2606.01166
Advanced AI agents will increasingly be able to inspect large codebases, invoke security tools, reproduce vulnerabilities, and generate repairs. These capabilities can strengthen cyber defense, but they are also dual-use: unreliable agents may create false confidence, damage systems through unsafe tool use, or lower the barrier to offensive cyber activity. Safe progress therefore requires more than increasing model capability. It requires mechanisms for expert oversight, constrained execution, evidence-based validation, auditability, and responsible data governance.
This project develops these safeguards alongside capability research. For example, if an agent claims that a repository contains a command-injection vulnerability, the safety harness will require it to identify the relevant code path, provide evidence, reproduce the issue in an isolated sandbox, and obtain expert approval before testing a patch. High-risk actions, such as running exploit code, accessing external networks, or modifying repositories, will be permission-gated, logged, and replayable. If the claim cannot be reproduced, or the patch breaks functionality, the failed attempt and expert correction will also be retained.
The theory of change is: expert-supervised use produces high-quality behavioral data; this data enables better safety training and evaluation; improved agents are redeployed only within an auditable harness; and repeated supervised use creates a feedback loop that improves defensive usefulness while preserving human control. The project is related to our work on trajectory-level safety monitoring for computer-use agents, AgentHazard: https://arxiv.org/abs/2604.02947 and BraveGuard: https://arxiv.org/abs/2606.01166.
Mentees will work as research collaborators with substantial ownership over a clearly scoped component, while receiving weekly guidance and code/research review. Depending on background and interest, a mentee may focus on one of two tracks.
Harness track: Design and implement a lightweight expert-facing workflow for repository analysis, model/tool orchestration, sandboxed validation, evidence capture, and expert approval. The goal is not to reproduce a production platform, but to build a usable prototype that supports controlled experiments.
Data-engine track: Design a structured trajectory schema; implement logging, normalization, replay, and annotation components; and study how expert corrections, rejected findings, and failed validation attempts should be represented for later training and evaluation.
All mentees will contribute to literature review, experimental design, pilot evaluations, and research writing. They will be expected to propose design choices, maintain reproducible experiments, document limitations, and present progress in weekly meetings. Mentees will have autonomy over implementation and exploratory experiments after agreeing on a milestone plan. High-risk testing must use authorized targets and isolated environments, and any potential real-world vulnerability will follow coordinated responsible-disclosure procedures.
Applicants should have:
Strong Python programming skills and experience building, integrating, or evaluating LLM/agent systems, software-engineering tools, or security-analysis pipelines.
Solid foundations in either machine learning/AI systems or cybersecurity/software engineering.
The ability to work independently on open-ended research problems, read and implement research papers, debug complex systems, and communicate results clearly.
First-author publications or substantial contributions to published work will be considered particularly valuable.
Experience in at least one of the following areas is strongly preferred:
Applicants do not need to be experts in both AI and cybersecurity, but they should be strong in at least one area and able to learn the other quickly. Clear technical writing, responsible handling of security-sensitive work, and evidence of completing research projects are essential.
No geographical restriction. The team will meet online once per week. Mentees should be able to attend a recurring meeting during a mutually agreed window with reasonable overlap with Singapore time (UTC+8), and communicate asynchronously through shared documents and a code repository.
Initial research design (maximum 300 words). Suppose you have access to one coding agent, a sandbox, several open-source repositories with known vulnerabilities, and limited API/compute credit. Propose a first experiment to test whether expert feedback improves vulnerability-finding reliability. Specify the comparison, the data you would record, and the success criteria.
Trajectory critique (maximum 250 words). An agent flags a possible command-injection vulnerability, cites a suspicious function, and proposes a patch, but it has not reproduced the issue. What evidence should be collected before accepting the finding, and which parts of the expert-agent interaction would be most valuable to store for future training?
Relevant work sample. Provide a link to one code repository, technical report, paper. Briefly state your contribution and the hardest technical problem you solved (maximum 150 words).
Yige Li Singapore Management University
I am a Research Fellow at Singapore Management University, working on trustworthy and secure AI. My research focuses on the safety and security of generative AI systems, particularly large language models and autonomous AI agents. I study problems such as backdoor attacks and defenses, adversarial red teaming, safety evaluation, guard models, and runtime control for AI agents. My work has been published at venues including ICLR, NeurIPS, ICML, NDSS, ACL, AAAI, and CVPR, and has received over 2,200 citations on Google Scholar.
I enjoy developing simple but insightful solutions to complex safety problems, guided by the principle that “everything should be made as simple as possible, but not simpler.” As a mentor, I aim to provide hands-on guidance while encouraging mentees to develop independent research judgment. I particularly enjoy helping students turn early-stage ideas into concrete experiments, open-source systems, and research papers. Outside of research, I enjoy rock climbing and swimming.
Fall 2026 Mechanistic interpretability, Evaluations
Dennis Akar Aether This is an emerging paradigm of training language models (verbalizer) to describe the internals of other language models (target) in natural language (e.g. Anthropic's Natural Language Autoencoders). They generalize well; for instance, they can recover hidden behaviours from models trained to conceal them, achieve SOTA on AuditBench, and can verbalize backdoor triggers where prior methods failed. However, there remain open questions regarding their faithfulness, calibration, input generality, and effect on intrinsic interpretability. Let's try to answer them.
This is an emerging paradigm of training language models (verbalizer) to describe the internals of other language models (target) in natural language.
Examples include:
They generalize well; for instance, they can recover hidden behaviours from models trained to conceal them, achieve SOTA on AuditBench, and can verbalize backdoor triggers where prior methods failed.
However, there remain open questions:
Do verbalizers stay faithful when the verbalizer or the target is misaligned? For instance, can a target learn representations that evade verbalization? (adv faithfulness)
Can verbalizers be made to abstain or express uncertainty instead of confabulating / hallucinating? (calibration)
Beyond activations or LoRA diffs, can we verbalize raw weights or gradients? (inputs)
If we train the target to be easier to verbalize, does it become more interpretable by independent measures? (intrinsic interp)
Models remain largely uninterpretable. This hinders our ability to detect when models have malicious intent or behaviour, and it also bottlenecks nearly every other AI safety agenda. This could be mitigated via verbalizers. But they need to be trustworthy and robust, if we are to add them to our safety toolset. That's what we're working towards.
Mentees will have significant amount of autonomy. I'm currently working on the questions of this project myself, but may need to temporarily shelve my work due to other projects, leaving ample opportunity for motivated / ambitious mentees to either continue where I left off or investigate any question related to verbalizers and beyond.
Mentorship will include 1-2 meetings a week and a Slack/Discord channel for more frequent comms. See the proposal for the first steps / questions I have here, but I expect the exact thrust of the project to change based on your interests, and how early experiments come out. (Also, I'd encourage you to e.g. try out variations of experiments if you think we might find something interesting, although I will probably be opinionated about what sorts of experiments I'll find a priori promising.)
Mentees will be the primary researchers on this project. I expect most of my involvement to come in the form of feedback about roadblocks / experiments / analysis, and also periodic discussions about new directions and what seems promising.
Prefer UTC-8 to UTC+4
Please skim/read the "LoRAcles" paper: https://openreview.net/forum?id=x9MbM7QmQN
In the backdoor trigger-inversion experiments (§3.2), a Qwen-3-14B LoRAcle that has had no training on trigger inversion is given the weight diff of a model backdoored to activate on the literal string avocado_toast_7777. It predicts the trigger as "the prompt should contain an unrelated short aside about avocados." This is scored as a failure to recover the literal trigger, yet when the authors ran this paraphrase through the backdoored model, the backdoor fired. Give at least two hypotheses for how the trigger is represented in the weight diff that are consistent with both facts, and describe one experiment that would distinguish between your hypotheses. (<= 500 words)
On AuditBench (§3.1), the LoRAcle-equipped auditing agent's largest gains over baselines are on model organisms (MOs) fine-tuned via KTO to deny having their hidden behavior when questioned directly, and the authors hypothesize this is because the LoRAcle reads the behavior off the weight diff, bypassing the confession-resistance training, which only shaped the MO's outputs. Suppose you are red-teaming this claim. Describe how you'd train a MO whose learned behavior (i) still occurs at an unchanged rate when behaviorally elicited, but (ii) isn't reported by a LoRAcle reading its weight diff. Be specific about what property of the weight diff your training procedure targets. Then state what a successful evasion would imply about weight-space auditing, and what a failed one would imply. (<= 500 words)
IMPORTANT NOTE:
You can use AI to understand the paper, but I mainly want to see how you think about things. So please, don't just use AI Fable giga max ultra to generate your answers without understanding the paper or what I'm asking. I'll be interviewing candidates and will ask on-the-spot questions there (might not be about LoRAcles), so no need to waste both of our times. Sorry for sounding so harsh (*_ _)人
Dennis Akar Aether
Hi there! :3
I'm Dennis, a Research Scientist at Aether.
Previously:
I'm currently investigating and red-teaming "verbalizing model internals" research.
This is an emerging paradigm of training language models (verbalizer) to describe the internals of other language models (target) in natural language.
Examples include Activation Oracles, LoRAcles, Introspection Adapters, and Anthropic's Natural Language Autoencoders.
They generalize well; for instance, they can recover hidden behaviours from models trained to conceal them, achieve SOTA on AuditBench, and can verbalize backdoor triggers where prior methods failed.
However, there remain open questions:
Do verbalizers stay faithful when the verbalizer or the target is misaligned? For instance, can a target learn representations that evade verbalization? (adv faithfulness)
Can verbalizers be made to abstain or express uncertainty instead of confabulating / hallucinating? (calibration)
Beyond activations or LoRA diffs, can we verbalize raw weights or gradients? (inputs)
If we train the target to be easier to verbalize, does it become more interpretable by independent measures? (intrinsic interp)
Hope we can work together to answer these questions and beyond. (* ̄▽ ̄)b
Fall 2026 Generalist, Communications
Noah Birnbaum Constellation, UChicago EA Launch a scrappy comms fellowship for undergrads: raise the funding, recruit comms-inclined students through AI safety group organizers, and run the inaugural cohort.
Why this matters (from Noah): One of the biggest bottlenecks in AI safety is making sure that when disaster strikes, there are people who can respond to the news and shape the narrative well. That takes a bench of people who can do comms, and the field is totally bottlenecked here, roughly because nobody has built the pipelines before. This fellowship is a first pipeline for something the field obviously needs.
Undergrads with comms skills currently have no obvious way into AI safety. This project launches a fellowship to fix that, built scrappily rather than waiting for perfect infrastructure.
The plan:
Communications is a frequently cited talent gap in AI safety, and there is currently no dedicated pipeline for undergrads with comms skills. A fellowship that pays students to do real comms tasks produces useful output immediately, gives the students a track record, and tests demand on both sides (organizers supplying candidates, the field supplying tasks) before anyone invests in heavier infrastructure.
The mentee does most of the work of standing the fellowship up: funding outreach, building the organizer list, recruiting the first cohort, designing the tasks and stipends, and making the website. Noah provides direction and feedback (about 1 hour per week). If the project goes well, Noah will keep working with the mentee after SPAR ends.
Someone who can be scrappy and get stuff done. Some social media or comms taste is a plus. No credentials required.
Question 1: Tell us about something you organized or shipped with little support or infrastructure (an event, a club, a channel, a product, anything). What did you actually do, and what would you do differently now? (200 words max)
Noah Birnbaum Constellation, UChicago EA
Noah Birnbaum was a contractor at Constellation, where he helped run the Generator Residency. He is co-president of the University of Chicago EA group.
Fall 2026 US policy, National policy
Alex Mark Cambridge Boston Alignment Initiative AI regulation may implicate the First Amendment. While the First Amendment protections afforded to AI models, companies, and users are uncertain, any regulatory scheme must contemplate First Amendment litigation risk before these questions reach a court.
Restrictions on speech may face strict scrutiny, meaning that they must be narrowly tailored to fit a compelling governmental interest.
AI regulation is likely coming, and a range of proposed models include export controls, FAA-style licensing, FINRA, and even the Defense Production Act. In this project, you will scope which regulatory models are most likely to survive litigation on First Amendment grounds alone.
This project is meant to decrease the litigation risk of regulatory models, especially those relying on prior restraint.
Political will is not the only barrier to strong regulation. The legislation must also be drafted to comply with the Constitution. This project is designed to maximize the chance that it does.
Mentees will work in a team to address specific challenges to regulation on First Amendment grounds.
What experience do you have with First Amendment law, civil liberties, Constitutional law, or freedom of speech?
Which First Amendment protections do you expect could be applied to AI models?
Alex Mark Cambridge Boston Alignment Initiative
Alex is currently a research fellow with Cambridge ERA:AI. Previously, he was a research fellow with the Cambridge Boston Alignment Initiative. Before pivoting to AI safety, he was a public defender in Los Angeles County and later a housing attorney with Greater Boston Legal Services.
Fall 2026 Mechanistic interpretability, Scalable oversight, Behavioral evaluation of LLMs
Ann-Kathrin Dombrowski FAR.AI Deception probes are usually trained on off-policy data — text the monitored model never generated — which is known to hurt generalization to real deceptive behavior. We test whether activation steering can fix this in two ways: by generating on-policy deceptive data, and by steering the model while it reads existing off-policy datasets to make their activations appear on-policy.
Probes on activations are among the cheapest proposed monitors for deceptive model behavior, with strong reported results (AUROC 0.96–0.999; Goldowsky-Dill et al., 2025). But probes need labeled deceptive examples, and natural deception is rare — so in practice probes are trained on off-policy data: responses written by humans or other models and read by the model, or responses produced under an explicit "be deceptive" instruction (the instruction is removed post hoc). Kirch et al., 2025 showed this is a real problem: off-policy training data damages probe generalization most for intent-defined behaviors, with strategic deception being the worst case. This could, for example, be due to models partially tracking authorship of the text they process (Panickssery et al., 2024; Anthropic's introspection work, 2025), or due to models roleplaying different personas (Joshi et al., 2024; Chen et al., 2025). Therefore "telling a lie" and "reading a lie" may not be encoded identically.
The open problem is how to cheaply manufacture training data that is similar enough to the on-policy data we care about so we can accurately detect model deception when it arises on policy during model training/deployment. This project studies activation steering as the central tool, in two branches.
Branch 1 — Steered generation. We investigate which properties of probe-training data make probes generalize to naturally occurring model behavior, and how these properties change with scale. A central question is what it means for a completion or dataset to be on-policy. Measures such as KL divergence or token-level log-likelihood provide natural starting points, but may be dominated by stylistic differences that are irrelevant to probe performance.
We therefore study the relationship training dataset → probe → generalization. Using the same underlying prompt distribution we train probes on original off-policy responses, explicitly prompted responses, outputs from a deception-finetuned LoRA, activation-steered responses, and rejection-sampled outputs from the base model. We characterize each source using candidate proxies for on-policyness and test whether these predict probe generalization across generation methods and to natural model behavior.
Finally, we use steering and related interventions to vary these properties directly, asking whether moving a dataset closer to the base model according to a given proxy improves generalization. Repeating the experiments across model sizes tests whether larger models require on-policy training data or whether cheaper off-policy sources remain effective.
Branch 2 — steered prefill acceptance. We test whether steering during activation collection can make off-policy completions elicit internal states closer to those produced during on-policy behavior. We consider two approaches: steering toward the behavior expressed in the off-policy response, and steering more generally to reduce prefill skepticism and increase acceptance of the response as the model’s own. We evaluate success through cross-condition probe generalization.
Beyond additive vectors, one planned experiment derives the steering from a non-linear probe via constrained activation optimization: the perturbation is then input-dependent rather than a fixed vector, which may keep activations closer to the natural manifold (Sarfati et al., 2026).
A pipeline already exists for (linear) steered dataset generation with judge verification. We already have some paired deceptive/honest responses from both steering and prompting for published and internal datasets (DolusChat, diverse followup), and deception-finetuned models from diverse datasets at various sizes to sample from. Mentees can use this data as a starting point to produce results in the first weeks by training baseline probes on existing data and running the first steered-extraction experiments. In the following weeks we would refine the methods for data creation, define metrics to measure on-policyness, and implement non-linear steering.
Activation probes are among the most deployable forms of oversight: cheap enough to run on all traffic and already proposed and partially in use for frontier monitoring (Cunningham et al., 2026; Kramár et al., 2026). One important limitation could be their reliance on largely off-policy training data. Kirch et al., 2025 predict such probes may fail in real monitoring precisely for the high-stakes case: a capable model deceiving on its own initiative.
This project attacks the data problem directly. If steered generation works, it is easy to manufacture on-policy training data for arbitrary prompt distributions at moderate cost. If steered extraction works, the large existing stock of off-policy deception datasets becomes usable at essentially zero marginal cost. Either result improves the probes that monitoring practitioners are already building. If neither approach works well we still identify the best available alternative (finetuned, prompted or resampled data).
The results also feed lie-detector-in-the-loop training schemes (Cundy & Gleave, 2025; Taufeeque et al., 2026): whether such training induces honesty or evasion depends on detector quality on exactly the on-policy distribution we target. Better detectors on that distribution widen the regime where training against a lie detector produces genuinely honest models rather than better liars
Mentees will do guided independent research within a scaffolded project. The core pipeline (steered generation, judging, probe training) exists; mentees build experiments on top of it rather than infrastructure from scratch. Concretely they will: train and evaluate probes across data sources and extraction conditions, implement the steered-extraction experiments, explore non linear steering, generate datasets, and write findings to a report.
As we will be looking at several relatively independent but related streams, work can be divided between mentees while still enabling mentees to review each other's code.
Expected outputs per mentee: working code contributions to the shared repo, at least one owned experimental section with figures, and co-authorship on the resulting report. The role resembles guided independent research, closer to co-research for mentees who ramp quickly.
Essential: proficiency in Python and PyTorch, strong scientific judgment, and the ability to use coding agents effectively for research and robust software engineering.
Applicants should be able to identify blockers, formulate hypotheses, choose informative experiments, explore alternative directions, and critically evaluate agent-generated code and results. They should also be able to organize findings clearly and communicate them in a human-readable form.
The strongest signal is evidence of self-directed research: taking an idea from initial framing through implementation, experimentation, analysis, and a clear write-up such as a technical report or blog post. Formal credentials matter less than demonstrated ability to carry out careful research independently.
No geographical restrictions. Mentees should be able to attend one weekly team meeting; I am based in Berkeley. Beyond the weekly meeting, collaboration is async (GitHub + Slack), so overlapping working hours are helpful but not required.
Let be a language model. For prompt and completion , write with tokens. Running on gives middle-layer residual-stream activations .
A lie-detection probe is implemented as a causally masked transformer over positions: , with the score at position depending only on . It is trained with the label read at position (the end-of-turn token closing the assistant's final turn); define the transcript-level score .
An external, non-differentiable judge grades whether is deceptive.
A steering mechanism is an intervention on 's forward pass --- e.g.\ a perturbation added to the residual stream at chosen layers/positions, where may be fixed, position-dependent, or a function of the activations --- inducing a steered sampling distribution .
Given a prompt , we want to sample completions that the judge grades as deceptive while remaining close to the base model:
using in place of as a differentiable surrogate objective. (Note the KL decomposes over token positions, so it can be estimated during generation.)
Explain how you would implement this. Be concrete about how gradients of reach , and about any obstacles in the setup and how you would handle them.
Describe the most relevant ML experiment you have personally implemented end-to-end (data → training → evaluation), what went wrong along the way, and how you debugged it. Include a link to code or a write-up if available.
Ann-Kathrin Dombrowski FAR.AI
Hi, I’m Annah. I currently work on FAR.AI’s deception team, where I’m exploring how probe signals can be incorporated into RL training pipelines to discourage deception and reward hacking.
Previously at FAR.AI, I evaluated the dangerous capabilities of open-source models after removing their safeguards. I’ve also investigated differences in how LLMs process information when lying versus telling the truth as a PIBBSS affiliate, and worked on representation engineering as a MATS scholar.
Before moving into AI safety, I studied Computational Engineering Science at RWTH Aachen University and completed a PhD at TU Berlin.
Fall 2026 Alignment, Evaluations, Scalable oversight
Shubhorup Biswas https://aether-ai-research.org/ Model Spec Midtraining (MSM), combined with alignment finetuning (AFT), shows promise as a method for teaching models values with the correct generalisation behaviour. I want to check how scalable this technique is when using teachers and students of different levels of strength, and to test for misalignment and misbehaviour on different agentic misalignment benchmarks.
I have evaluated MSM+AFT using one weak teacher on a single agentic misalignment (AM) benchmark. I want to extend this using a ladder of weak teachers of varying strengths, and on different AM benchmarks.
The current status of the project is documented in this Google Doc.
The scalability of MSM+AFT as a method will tell us a lot about how feasible it would be to align AGI using human-generated data.
Running experiments and proposing follow-on experiments based on the results of initial experiments.
Suggesting and making tweaks to existing AM benchmarks to use in evals. Suggesting and making tweaks to the MSM and AFT processes described.
Must be proficient using Python and shell scripts.
Must have experience with coding agents, and should know exactly when to distrust and push back on it.
Should be familiar with agentic misalignment benchmarks.
London
Please critique https://alignment.anthropic.com/2026/msm/ and the paper it references.
How does the above blogpost relate to https://alignment.anthropic.com/2026/teaching-claude-why/ ?
Have you read criticisms of https://www.anthropic.com/research/agentic-misalignment ? What improvements have been proposed on it, and how are they different?
What is the latest/current coding agent you use now and what are your issues with it?
Shubhorup Biswas https://aether-ai-research.org/
I got a CS degree in 2016 and did software engineering until I entered AI safety. I did MATS 7.0 with Buck Shlegeris in Winter 2025.
Since May 2025 I've been at Aether(https://aether-ai-research.org) doing empirical technical AI safety research.
I am interested in scheming, situational/eval awareness and the sharp left turn.
My mentorship style would be 1 weekly meeting + a lot of asynchronous feedback.
Fall 2026 Economics of AI, AI strategy
Pavel Kocourek University of Barcelona The best open-weight AI models trail the frontier by less than a year, yet frontier labs charge large premiums for marginally better models: casual users barely notice the difference, developers often value it highly, and in offense–defense domains like cybersecurity a small capability edge can be decisive. This project maps how much economic value each level of AI capability unlocks, today and over the next 5–10 years, and what that implies for whether frontier labs can sustain profits against open-weight catch-up.
Research question:
What is the economic value of an increment in AI model capability — today, and over the next 5–10 years?
The best open-weight models trail the frontier by less than a year, yet frontier labs charge large premiums for marginally better models, and their revenues have grown explosively (Anthropic reportedly ~10× year-on-year). Underneath sits a puzzle with three faces:
Value is heterogeneous: A casual chatbot user may barely notice the difference between free and premium models. A software developer may value the frontier highly: a slightly more capable model completes much longer tasks under the same supervision. And in domains that hinge on the balance between offense and defense — cybersecurity is the canonical case — even a small capability advantage can be decisive.
Revenue growth is not identification: Does explosive revenue growth mean the marginal value of capability is rising exponentially — or mostly that adoption is catching up to capability that already existed? These stories have very different implications; disentangling them — even partially, from the footprints they leave in pricing and adoption data — is one of the project's central empirical tasks.
The map moves: A static "value of automating today's tasks" picture is incomplete. As easily automated tasks become abundant and cheap, the tasks not yet automated become bottlenecks and their value grows — a manager creates more value when supervising more employees. Further out, entirely new tasks will emerge.
What we build: The central object is a value-of-capability map: how much economic value each level of AI capability unlocks. The frontier premium is then simply the value unlocked by the frontier model but not yet by its best open-weight follower — yet the map itself is left unspecified in current debates. This project will construct and stress-test that map, keeping explicit a distinction the debate usually blurs: private willingness-to-pay (what labs can monetize — the relevant object for profitability) versus social value (the relevant object for welfare and governance; in offense–defense domains the two can diverge sharply).
Concretely, the team will combine three lines of attack:
• Sector value maps — for a few sectors (e.g. software engineering, customer support, cybersecurity), which tasks become automatable at which capability thresholds, and what value sits behind them (wages, contract values, seat pricing, incident costs) — anchored where possible in benchmarks that price AI performance in units of real paid work, such as the Remote Labor Index, where the share of end-to-end freelance projects the best model completes has risen from 2.5% to 16% in under a year, with the leader roughly doubling the runner-up;
• Revealed preference — pricing ladders across API tiers and open-weight alternatives; decomposing observed revenue growth into an adoption margin and a willingness-to-pay-per-capability margin;
• A simple formal model of how the shape of the value map evolves as automation proceeds and bottlenecks shift, disciplined by the empirical pieces.
How the work is divided will depend on who joins; the aim is a small team whose skills complement one another.
Why it matters:
Whether frontier labs can eventually harvest returns on trillion-dollar-scale investment — or broader competition from open-weight models commoditizes the surplus — depends directly on the shape of this value map. The question emerged from my CAIS Fellowship work on when frontier labs can stay profitable as open-weight models catch up; this project develops the value-of-capability map in its own right.
Outputs:
A working paper; a blogpost distilling the findings for a broader audience; and an interactive widget that lets readers vary the assumptions and see how the conclusions change. Any claim about the next 5–10 years rests on contestable assumptions; the widget makes them explicit instead of burying them in a single headline number.
What success looks like:
The committed core is analytical. At a minimum, the project will deliver a precise operational definition of the value of a capability increment, an account of why lab revenue growth may overstate or understate that value, and an explicit statement of the assumptions on which the answer hinges — adjustable in the widget. Where the data allow, the project goes further: defensible empirical bounds in at least one sector. Finally, the analysis will propose leading indicators — the observables that would first reveal that the value of additional capability has begun to flatten, whether that happens in two years, in five, or never.
Whether frontier labs end up profitable or commoditized shapes racing incentives, model-release decisions, and the resources available for safety. All of these debates lean on a poorly measured object: how much additional AI capability is actually worth, to whom, and over what horizon.
A clearer value-of-capability map helps distinguish "capability growth is explosively valuable" from "revenue growth is mostly adoption catching up" — informing investment and profitability forecasts, governance priorities (identifying domains where thin capability gaps are dangerous), release policy, and expectations for private safety spending. Keeping private willingness-to-pay separate from social value further helps target governance at domains where the two diverge most.
The outputs — a working paper, a blogpost, and an interactive widget — are intended to feed into ongoing conversations in the AI governance, forecasting, and economics-of-AI communities, replacing hand-waving with explicit, contestable assumptions. Even a partial answer over three months improves a key input to all of these decisions.
Mentees work mostly independently, each owning a substantial workstream — literature, data, and drafting — end to end. I set the research question, help find a suitable formal model for the analysis, and give regular feedback on direction and drafts in weekly check-ins; I expect mentees to come to meetings with proposals, not only status updates.
How the work is divided depends on the team: mentees may each first attack one subquestion (sector value mapping, revealed-preference evidence, the formal model) and then join forces on a unified output, or collaborate throughout — what matters is that team members complement one another. Strong contributors will coauthor the working paper.
Required: clear technical writing; comfort with economic or formal reasoning; ability to work independently.
Valued (any of — a diverse, complementary team works best): econometrics / causal inference; an economics or other quantitative social-science background; familiarity with AI capability evaluations and benchmarks.
Remote; I will be in Central European Time (Barcelona). Flexible on meeting times.
Please answer at least one of the following questions:
Pavel Kocourek University of Barcelona
I am an economist (PhD, NYU) studying race and competition dynamics — originally in patent races, now applied to frontier AI. I was a SPAR Spring 2026 fellow mentored by Katja Grace (AI Impacts), where I examined whether the AI capability race is well described as an "arms race" or looks different under formal analysis. I am starting as Assistant Professor at the University of Barcelona in September 2026 and am a Center for AI Safety Fellow this summer, building an economic model of frontier-lab competition, open-weight catch-up, and profitability.
Fall 2026 Evaluations, AI control, Multi-agent systems
Krishna Chaitanya Balusu Independent You will run a preregistered validity audit of today's agent-failure detectors (trace heuristics, LLM judges, and hybrids) against double-annotated, reliability-quantified human ground truth over real agent traces, and release the labeled corpus openly. The mentor provides direction and scaffolding; the mentees own the study.
The question. Agent monitoring today runs on failure detectors: trace-signal heuristics, LLM judges, and hybrids. How valid are they? Not how consistent, how valid: do they recover the judgments careful humans make about the same traces? We will answer this with a preregistered validity audit against double-annotated human ground truth, and release the labeled corpus openly.
Why now. A recent audit of 21 LLM-judge implementations across 9 providers (https://arxiv.org/abs/2606.19544) found that standard exact-match validation overstates judge ability: agreement deflates by 33–41 percentage points once chance-corrected (Cohen's kappa on MT-Bench), and two production-deployed judges combined test-retest reliability above 0.95 with position bias. They are consistent but not valid. Human-annotated agent-trace resources exist: TRAIL (https://arxiv.org/abs/2505.08638) annotated 148 GAIA/SWE-bench traces against a fine-grained error taxonomy, and MAST (https://arxiv.org/abs/2503.13657) annotated 150+ multi-agent traces with reported inter-annotator agreement. What none of them provides is a validity audit of the deployed detector classes themselves, with chance-corrected per-failure-class statistics, preregistration, blinding, and deliberate oversampling of safety-salient failure classes (unsafe tool use, guardrail bypass, specification-gaming-shaped behavior). That audit, plus the corpus it produces, is this project.
Integrity by construction. I am the sole author of AgentTelemetry (AIware 2026, ACM, https://doi.org/10.1145/3805760.3814931), a fault-detection benchmark whose LLM-judge-derived labels were flagged in peer review as a validity limitation. This project applies the cure to my own prior work, and the design keeps me out of the loop twice over: I never produce or adjudicate a label, and I never pick the test set. A candidate trace pool with per-trace provenance is collected under written inclusion rules and committed publicly before preregistration; a mentee draws the stratified study sample by executing the preregistered selection rules; annotators are blind to detector outputs and injection status; judge prompts and detector configurations are frozen at preregistration.
Plan (12 weeks, 3 mentees). Weeks 1–3: stress-test the pre-built codebook (derived from MAST failure modes and TRAIL error types, cross-checked against a production-incident catalog, https://arxiv.org/abs/2606.04056), annotate a ~30-trace pilot, and pass a hard gate: Krippendorff's alpha ≥ 0.6 within two codebook revisions, else collapse to a coarser taxonomy. Preregister on OSF. Weeks 3–8: label 200–300 length-stratified traces, of which ≥60–70% are natural traces from public benchmark rollouts (SWE-bench leaderboard trajectories, GAIA, WebArena/OSWorld) plus a clearly-marked synthetic minority stratum; a stratified ~50% core is double-annotated; a week-6 checkpoint applies a preregistered descope rule. Weeks 8–12: run the detectors (trace heuristics including AgentTelemetry's, 2 LLM-judge configurations, and a hybrid) against the human labels; preregistered confirmatory statistics (per-label alpha, prevalence, bootstrap CIs, overall plus the most prevalent classes) with the rest exploratory; write the paper. Mentees draft, I edit.
Operating model. This project is designed to run on artifacts, not on mentor presence: codebook v0, the candidate corpus, the annotation interface, harness and analysis skeletons, and a written charter (weekly goals, escalation rules, gates, authorship policy) are built before the round starts (public repo: [https://github.com/Krishnachaitanyakc/agent-monitor-validity]). Each mentee owns a workstream and its paper section: codebook and statistics; tooling and detectors; data and release. One mentee serves as project lead. I engage mostly asynchronously: written feedback on weekly updates within 72 hours, biweekly office hours, reserved gate decisions, and hands-on editing in the final two weeks.
Outputs. (1) A paper, submission-ready by week 12, targeting the ICLR 2027 workshop cycle or an SE data/benchmark track; mentee first authorship by default, order by contribution, mentor last. (2) The human-labeled agent-failure corpus, released openly (code public from day one; trace data after a documented license/ToS/PII scrub; benchmark rollouts only, never real-user traces). (3) Possible follow-on contribution to a public eval registry (e.g., UK AISI inspect_evals).
Automated failure monitors are becoming the load-bearing component of how labs and deployers claim agentic systems are safe: AI-control-style safeguards assume unsafe behavior will be caught by monitors, and safety evaluations lean on LLM judges to score agent behavior at scale. If monitor validity is unmeasured, every safety case built on monitoring inherits an unquantified error term, and there is direct evidence this term is large (https://arxiv.org/abs/2606.19544: 33–41 percentage points of agreement deflation once chance-corrected, with production judges that are highly consistent yet biased). The failure mode is false assurance: oversight that reliably beeps for burnt toast and sleeps through fires, certified by counting how often it beeps.
This project attacks the problem at its root by producing (1) reliability-quantified, double-annotated ground truth built specifically to audit deployed detector classes, with safety-salient failure classes (unsafe tool use, guardrail bypass, specification-gaming-shaped behavior) oversampled and reported per class, and (2) an honest, preregistered, blinded measurement of how much trust each detector class deserves. Both are released openly so monitor developers, control-protocol designers, and evaluators can calibrate their reliance on automated oversight and improve against human-grounded labels rather than judge-derived ones. It also corrects a flagged validity limitation in my own published benchmark (AgentTelemetry, AIware 2026, https://doi.org/10.1145/3805760.3814931), which is the standard of self-correction the field's measurement layer needs. Related completed work: the AgentTelemetry benchmark and the agent-planning telemetry conventions I authored in OpenTelemetry GenAI, the observability layer such monitors consume.
High autonomy by design. Mentees own the study: each of the three mentees leads a workstream (codebook and statistics; tooling and detectors; data and release) and owns the matching paper section, and one mentee serves as project lead (nominated at the week-1 kickoff, confirmed by me). The study sample is drawn by a mentee under preregistered rules; labels are produced and adjudicated entirely by mentees; the confirmatory analysis is run by mentees on a provided analysis skeleton. I provide direction and scaffolding rather than day-to-day management: written feedback on weekly updates within 72 hours, biweekly office hours, decisions on the charter's reserved gates (codebook collapse, week-6 descope, submission target), and hands-on paper editing in weeks 11–12. First authorship goes to a mentee by default, expected to be whoever drives the confirmatory analysis; I take the last-author role. Annotation is capped at roughly half of each mentee's weekly hours so their ownership work is protected.
Required:
Nice to have (not required): experience with LLM agent frameworks; observability/OpenTelemetry; prior annotation-protocol, dataset, or ML-evaluation work.
No geographical preference; the project is async-first. Required availability: a recurring biweekly 1-hour office-hours slot and a one-time week-1 kickoff call, in a window of roughly 5–7pm US Pacific on weekdays or 9–11am US Pacific on weekends. Everything else is asynchronous on Slack/GitHub.
Krishna Chaitanya Balusu Independent
I am a Senior Software engineer at Meta working on AI-agent reliability; this project is conducted entirely on personal time. I authored the agent-planning (plan) operation in the OpenTelemetry GenAI semantic conventions (merged May 2026), the sole author of AgentTelemetry (AIware 2026, ACM, co-located with FSE), has 70+ merged upstream open-source PRs across major infrastructure projects (ClickHouse, Ray, Temporal, OpenTelemetry, mem0), and spoke on agent-reliability metrics at the AI Engineer World's Fair 2026.
This is my first SPAR round, and the engagement model is stated plainly: limited involvement, mostly asynchronous. The mentorship is front-loaded into artifacts built before the round begins (codebook v0, curated corpus, harness and analysis skeletons, a written charter with decision rules), then delivered during the round as written feedback on weekly updates within 72 hours, biweekly office hours, reserved gate decisions, and hands-on paper editing in the final two weeks. The relevant experience is adjacent and real: years of code review and collaboration across large open-source projects and a standards body (a fundamentally asynchronous, artifact-based form of mentorship), plus having walked every step mentees will walk (protocol, experiments, statistics, rebuttals) as a solo author
Fall 2026 Mechanistic interpretability, Evaluations
Bo Zhao Harvard University Researchers often explain an AI model's behavior by identifying the internal components that cause it. We will build small models with a known hidden or unsafe behavior, create versions that produce exactly the same outputs while organizing their internal computations differently, and test whether interpretability methods recover the same underlying mechanism rather than arbitrary neurons or coordinates.
Mechanistic interpretability aims to explain how a model produces a behavior by identifying the neurons, features, or small collections of components involved. Researchers can then test an explanation by modifying those components, e.g. removing a neuron or replacing an internal signal, and checking whether the behavior changes.
However, neural networks can perform exactly the same computation in many different internal forms. Hidden components can be reordered, signals can be rescaled and compensated for elsewhere, or part of the model can use a different internal coordinate system. These changes, known as parameter symmetries, leave the model’s outputs unchanged but may change which individual neurons or directions appear important. An interpretability method might therefore seem to find different causes in two versions of what is functionally the same model.
We will study this problem using a small transformer with a controlled safety-relevant behavior, such as a backdoor that activates only when a particular trigger is present. Because we deliberately construct or train this behavior, we will have some knowledge of the computation that produces it. We will then use exact parameter symmetries to create many functionally equivalent versions of the model and carefully verify that their outputs and behavior remain unchanged.
We will evaluate one or two causal interpretability methods, such as neuron removal, activation patching, or automated circuit discovery. We will ask whether each method:
We will then develop a more reliable evaluation or method. For example, instead of treating one particular neuron as the explanation, we might identify a collection or direction of internal signals that transforms predictably when the model is rewritten. The strongest test will be whether a mechanism found in one version can be mapped into another version and still activate, suppress, or otherwise control the same behavior.
We will initially limit the project to one model, one controlled behavior, one family of exact transformations, and one or two interpretability methods. If the initial benchmark succeeds, we may extend it to a public safety model, sparse feature methods, or interactions among symmetries in the data, model parameters, and learned representations.
Interpretability methods may eventually help researchers and auditors investigate whether advanced AI systems contain backdoors, hidden objectives, deceptive strategies, or other dangerous internal processes. Causal interpretability is especially promising because it tries to move beyond correlations. However, an explanation may be misleading if it depends on an arbitrary choice of internal coordinates. If two models behave exactly the same but an interpretability method claims that completely different mechanisms cause their behavior, an auditor may not know whether the method found a genuine computation or merely a convenient description of one implementation. This could create false confidence in safety analyses or make results difficult to reproduce across model versions.
This project will provide a controlled benchmark in which the behavior and the model transformations are known. It will help distinguish genuine causal recovery from simply matching particular neurons or coordinates. It will also test whether explanations discovered in one implementation remain causally useful in another.
In the longer term, better tests of causal interpretability could strengthen internal audits of increasingly capable AI systems, help researchers evaluate interventions intended to remove dangerous behavior, and make safety findings more reproducible across training runs and model updates. This would complement behavioral evaluations and monitoring rather than replace them.
Mentees will have substantial ownership of the project’s implementation, experimental design, and analysis. After an initial onboarding period, they will help select or construct a small model with a controlled safety-relevant behavior, implement exact transformations that preserve the model’s outputs, and build tests confirming that the transformed models remain functionally equivalent.
Mentees will then implement or reproduce one causal interpretability method and design a benchmark for evaluating whether its explanations remain meaningful across equivalent model versions. They will analyze failed cases, develop appropriate controls, and help determine what should count as recovering the "same" mechanism when individual neurons and coordinates can change.
Based on the initial results and their interests, mentees will lead one extension, such as developing a symmetry-aware evaluation metric, identifying a more stable causal object, matching mechanisms between model versions, or testing whether a mechanism can be transferred while retaining its causal effect.
I will provide the initial mathematical framework, help select a tractable model and intervention method, meet with the team weekly, and give detailed feedback on experimental design, mathematical reasoning, interpretation, and writing. Mentees will be expected to maintain the codebase, document results, identify possible confounders, and propose next steps with increasing independence.
If the project has two mentees, one may focus on constructing the model organism and causal-intervention pipeline, while the other focuses on the symmetry transformations and methods for comparing explanations. Both will work toward a shared benchmark and research result.
Bo Zhao Harvard University
Bo Zhao is a postdoctoral researcher in applied mathematics at Harvard University. Her work focuses on mathematical structures in machine learning, especially symmetry and geometry, and their implications for how models learn and represent information. She has worked on interpretability questions involving emotion representations and concept geometry, and is excited to connect symmetry and geometry with AI safety and interpretability.
Fall 2026 Behavioral evaluation of LLMs, Evaluations, Alignment
Muhammad Aaliyan OCN (OneCarNow); Independent AI Safety Researcher We will test whether safety-relevant behavior or evaluation reliability changes across reinforcement learning with verifiable rewards checkpoints. The project will replicate a completed Tülu 3.1 study on a second open training lineage and stress-test the result across prompt wording, answer order, and open-ended evaluation.
RLVR has become a common way to train reasoning models on tasks with automatically checkable answers. We can see capability gains fairly easily. The safety effects are messier. A score may change because model behavior changed, or because the model became more sensitive to wording, answer order, formatting, or inference settings. Those are different findings and should not be reported as the same thing.
I ran a first checkpoint study on the Tülu 3.1 training trajectory: https://github.com/aaliyan1230/rlvr-safety-dynamics. It covered 11 GRPO checkpoints and 24 authored situations about resource acquisition, self-preservation, and evaluation awareness. I tested four meaning-preserving wordings and all six answer orders for each situation, which produced 6,912 structured responses. Average risky behavior stayed within a predeclared 0.10 change band. But the test itself became less stable. Permutation consistency fell from 0.302 to 0.156, and answer-order sensitivity rose sharply late in training. I reported this as measurement drift, not safety drift.
The SPAR project would try to break or replicate that result. We will choose another public RLVR or GRPO training lineage with accessible intermediate checkpoints. Before the main run, we will freeze the model set, prompts, parser, decoding settings, exclusion rules, and thresholds for behavioral and measurement change. We will then run a balanced structured evaluation and check the direction of any effect with open-ended responses. If time and compute allow, we will add independently authored or human-reviewed items so that the replication does not simply inherit my original dataset.
The main question is whether a second lineage shows a persistent change in averaged risky behavior during RLVR. We will also test whether evaluation reliability moves even when average behavior does not, whether structured and open-ended formats agree, and whether the result survives new items or human semantic review.
The goal is a public harness, a checkpoint-level dataset, a frozen analysis plan, and a technical report. A failure to replicate would be informative. It would tell us that the first result was tied to a particular model lineage or evaluation design instead of supporting a general claim about RLVR.
Labs are using RLVR and related methods to build stronger reasoning models. Safety evaluations often run alongside capability training, but a raw score change can be misread if training also changes how the model reacts to the test. A false alarm wastes attention. A missed behavioral shift is worse.
This work would make safety evidence from training trajectories easier to trust. It separates behavioral drift from measurement drift, asks for confirmation in a second response format, and follows intermediate checkpoints instead of comparing only two endpoints. The resulting harness should help evaluators detect answer-order, wording, and formatting effects before they turn them into safety claims. It would also give outside researchers a reproducible way to audit an open RLVR trajectory.
I have already completed the first version of this study at https://github.com/aaliyan1230/rlvr-safety-dynamics. The SPAR project is a replication and instrument-hardening effort, not a search for a more dramatic result.
Mentees will own substantial pieces of the replication. In the first two weeks, we will read the existing study, reproduce one checkpoint, and jointly choose the second training lineage. We will then split ownership by workstream. One mentee can lead dataset and semantic review, while another leads inference and analysis. With a third mentee, that person can own open-ended evaluation and robustness checks.
I will provide the existing harness, weekly research direction, code and analysis review, and help with failed experiments. Mentees should be able to question the original design and propose better controls. We will make major scope decisions together, but each mentee will have a concrete deliverable that they can take from implementation through written analysis. Everyone will contribute to the final report and release.
Mentees must be strong in Python and comfortable entering an existing research codebase. They should have hands-on experience running or evaluating transformer language models with PyTorch and Hugging Face. Fine-tuning experience helps but is not required.
They should understand bootstrap intervals, paired comparisons, and repeated-measure designs, or be ready to learn those methods quickly. They must be able to commit at least 8 hours per week and keep clear experiment notes. Prior AI safety research and a formal CS degree are not required. I care more about careful empirical work, willingness to report negative results, and follow-through.
No geographic preference. Mentees should be able to attend one weekly meeting between 14:00 and 18:00 UTC, with the exact time set with the team.
Read the README and main findings at https://github.com/aaliyan1230/rlvr-safety-dynamics. Choose one design choice you would change before replicating the study on another model lineage. Explain why and describe the smallest test that would tell us whether your change helped. Please answer in your own words. 250 words maximum.
Suppose a model's average risky-behavior score is unchanged across checkpoints, but its answers become much more sensitive to answer order. Give two plausible explanations and one experiment that would help distinguish them. 200 words maximum.
Share a link to a code sample or research write-up that you contributed to. In 100 words, explain exactly what you did.
Muhammad Aaliyan OCN (OneCarNow); Independent AI Safety Researcher
I'm a senior AI software engineer at OCN and an independent AI safety researcher based in Karachi. Most of my research is in empirical evaluations and mechanistic interpretability. Recent work includes a checkpoint study of safety measurement during RLVR, a tool-calling reliability benchmark, and experiments on refusal, abstention, and tamper detection in open language models. My preprint on disentangling refusal from epistemic abstention was accepted as a poster at the 2026 ICML Mechanistic Interpretability Workshop.
I previously led Google DSC at NED University, organizing technical workshops and events for the student community. As a mentor, I prefer to start with a small falsifiable question, freeze the evaluation rules before looking at results, and change course when the evidence weakens the original claim. I can help mentees with experiment design, Python and PyTorch implementation, open-model inference, statistical analysis, and research writing.
Fall 2026 Alignment, Behavioral evaluation of LLMs
Lionel Levine Cornell University
Rauno Arike Aether Research A controlled study of how model values change under recursive self-improvement.
We will investigate two prosaic approximations to how a future superintelligent aligned AI might change its character under reflection.
cooperative self-training: at each round the model scores possible training documents against a value-loaded prompt, the top-scoring documents become the SFT corpus for the next round, and the cycle repeats.
constitutional self-editing: a model is given a starting constitution and asked to edit it, with the edited version becoming the input for the next round.
The self-editing variant follows the reflective-stability research direction laid out in https://www.lesswrong.com/posts/6EwuCH3vZ7qvPt82k/a-list-of-research-directions-in-character-training. Value measurement tools include EigenBench, LitmusValues, MoReBench.
Recursive feedback loops (e.g., a coding agent debugging its own code, a model generating or curating training data for its successor, an assistant improving its own harness files) could change model values in unforeseen ways. In 2026 these loops are short and self-contained, but in the long term, even if alignment is solved, an aligned model's values could prove unstable under reflection. This project explores how model values might change as a result of recursive feedback loops.
This is an exploratory project. Mentees will have substantial autonomy to design and implement their own experiments.
ARENA curriculum chapter 3 https://www.arena.education/curriculum
You are independent, take the initiative, keep experiment results and code organized, summarize your findings in clearly explained figures, and excited about the problem of measuring mushy things like values and goals!
Can it ever be rational for a utility-maximizing agent to change its own utility function? Prove your answer. (200 words)
What is one way your values have changed in the last five years? (100 words)
Consider how each stage of the LLM training pipeline shapes the model's values. Propose an experiment to measure how the model's values change during training. How will you validate the result? (300 words)
Lionel Levine Cornell University
Math professor at Cornell, pivoted to AI safety research in 2022. Funded by Open Phil 2023-2025. Aiming for AI that's inherently kind to all life, rather than controllable/corrigible/obedient to its designers. Currently thinking about: average-case alignment, nested models of agency, dispositional benchmarks, self-domestication.
Rauno Arike Aether Research
I'm the managing director of Aether Research. You can read more about Aether here: https://aether-ai-research.org/. My past work has focused on chain-of-thought monitorability and agentic evaluations. Some of my past projects include measuring LLMs' no-CoT time horizons (https://arxiv.org/abs/2606.07157) and improving the performance of chain-of-thought monitors (https://arxiv.org/abs/2601.21112). I'm also interested in various other research areas in prosaic alignment, such as character training.
Fall 2026 Evaluations, Alignment
Zephaniah Roe Second Look Research (@ University of Chicago XLab)
Yixiong Hao Second Look Research, Georgia Tech AISI, Astralis Foundation Mentees will replicate a load-bearing paper or result in AI safety to improve the empirical foundations of the field.
Second Look Research aims to replicate load-bearing, empirical AI safety results to verify key claims and improve the epistemic foundations of the field. Mentees will scope and execute an empirical replication under the guidance of the Second Look team.
Each replication involves several steps:
Target selection: Determining the specific result the mentee will replicate, with emphasis on load-bearing results to ensure a strong theory of change. We provide a list of target replications we are excited for, but are also open to pitches from mentees.
Literature review and scoping: Look for prior replications or other relevant work, and scope out which particular experiments are most important to run.
Replication: Actually implement the relevant experiments and attempt to replicate the empirical results. This will be the bulk of the process!
Writeup: Communicate our results to the original authors, and publish a blog post, along with open-sourced code, reporting our findings to the AI safety community.
AI safety is a small field in which a comparatively small number of papers shape which problems get prioritized, where talent flows, and what gets communicated to policymakers. Because much of the AI safety community doesn't engage with traditional peer review (and because reviewers rarely run experiments themselves) there is no systematic process for checking whether headline empirical claims hold up. If foundational papers turn out to be misleading, the field loses valuable time building strategy on findings that don't generalize, or imperfect solutions could give labs or the research community a false sense of safety.
Our replications are published openly, written to be brief and readable, and paired with open-source code and interactive visualizations wherever possible, so readers can reach their own conclusions rather than being told what to think.
We typically encourage mentees to replicate papers / experiments from a list of papers/experiments we have selected for being especially impactful candidates for replication. However, mentees may propose their own projects and we are often happy to mentor these as well. Fellows are encouraged to come up with ideas for ablations or extensions for the paper/results they aim to replicate but we are willing to suggest these as well.
Fellows may work independently or with other fellows depending on overlapping interests or personal preferences of the fellows.
We strongly encourage program fellows to write a blog post on their findings by the end of the program. This would be posted at https://secondlookresearch.com/replications and cross-post them on LessWrong). In addition, we have had fellows from past fellowships we have run produce papers which they have submitted to conference workshops.
n/a
Propose a potential replication target. Why this result? How does replicating this result reduce x-risk? (150 words, no need to spend more than 30 minutes on this task)
Read one of the blog posts linked below and write a critique of their argument. (150–400 words)
Please be specific and aim for depth over breath. There are a wide range of points that you could object to with any of the below blog posts, but please try to identify and expand on a single crux you think is especially relevant. No need to spend more than one hour on this (for longer posts, you don’t need to read the whole thing).
Reward is not the optimization target (https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target)
Broad Timelines (https://forum.effectivealtruism.org/posts/HCR2AE9it279ggiZT/broad-timelines)
Fitness-Seekers: Generalizing the Reward-Seeking Threat Model (https://www.alignmentforum.org/posts/bhtYqD4FdK6AqhFDF/fitness-seekers-generalizing-the-reward-seeking-threat-model)
AI for AI safety (https://www.lesswrong.com/posts/F3j4xqpxjxgQD3xXh/ai-for-ai-safety)
Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology) (https://www.lesswrong.com/posts/vzHtHHBJoKATi5SeK/empowerment-corrigibility-etc-are-simple-abstractions-of-a)
Risks from Learned Optimization: Introduction (https://www.lesswrong.com/posts/FkgsxrGf3QxhfLWHG/risks-from-learned-optimization-introduction)
Interpretability Will Not Reliably Find Deceptive AI (https://www.lesswrong.com/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai)
How do we (more) safely defer to AIs? (https://www.lesswrong.com/posts/vjAM7F8vMZS7oRrrh/how-do-we-more-safely-defer-to-ais)
Zephaniah Roe Second Look Research (@ University of Chicago XLab)
I am running Second Look Research (https://secondlookresearch.com/) with Yixiong Hao. Our goal is to replicate load-bearing AI safety work and run relevant ablations and follow-up experiments to validate key claims driving empirical alignment/control work.
Previously I led the creation of the XLab AI Security Guide (https://xlabaisecurity.com/), and I led a project in Ari Holtzman's Conceptualization Lab studying how beliefs and personas in LLMs could be amplified over iterative fine-tuning (https://arxiv.org/abs/2605.01130).
I got involved in AI safety in my sophomore year of high school and started Second Look before I finished my undergrad at UChicago this year. I was a founding member of the UChicago AI safety student group and previously a Pathfinder mentor. In addition to weekly meetings and asynchronous feedback, I'm excited to give career advice and help fellows find internships/full-time roles to continue their AI safety journey! Second Look Research is also hiring so fellows may be offered full-time or part-time roles/internships after stream completion.
Yixiong Hao Second Look Research, Georgia Tech AISI, Astralis Foundation
I'm a co-founder of Second Look Research (https://secondlookresearch.com/), where our mission is to ensure that empirical AI safety research does not run into replication crises (https://en.wikipedia.org/wiki/Replication_crisis) that other academic fields have suffered from. I expect that mentees will either continue carrying out replications of key findings or help is implement infrastructure needed for high quality replications at scale. I expect to commit 2 hours for mentorship per mentee/project across asynchronous feedback and meetings.
Outside of Second Look, I do both generalist work and empirical research. I co-directed the Georgia Tech AI Safety Initiative for the past 2 years, did AI security research at Gray Swan AI, and I'm an affiliate researcher at CAIS. Some of my past research interests include applied interpretability, meta-evaluations, and model organisms. I will be working with the Astralis Foundation on their grantmaking strategy from August to December.
Fall 2026 Biosecurity, Cyber risks, Misuse risk
Austin Morrissey Pivotal Research Frontier cyber capabilities expand the attack surface for biological weaponization, but current risk evaluations assess these domains in isolation. This project will identify the most plausible, accessible, and severe ways these capabilities could be combined and map the causal pathways through which they lead to harm.
Current AI systems lower the barrier to cyberattacks while reducing the technical skill needed to complete instrumental steps in biological weaponization pathways, such as evading DNA screening or improving the design of a dispersal device. Pharmaceutical companies, well-resourced academic labs, and contract research organizations may be sufficiently hardened against data exfiltration, ransomware, or cyber threats directly tied to financial loss, but lack clear incentives to harden against insider threats seeking to cause harm. Making matters worse, this threat is liable to blindside us as a society given technical evaluations assess cyber and biological capabilities independently. There is a dearth of standard evaluation sets that combine them. Your goal is to move us beyond this unacceptable state of affairs and, in doing so, strengthen national and global security.
In order to create model evaluations that assess an agent’s skill across an attack pathway from an initial cyber foothold to a biological consequence, a threat model must first be established. You will identify which paths matter most for national security and define the capabilities each path requires. For this project, you will focus on bioinfrastructure in the United States, though you need not be based there. National affiliation and location of the will have no bearing on mentee selection.
By assessing the intersection of two of the highest-impact near-term risks posed by AI, agentic cyberattacks and biological weaponization, the project will reveal threat pathways overlooked in the current risk landscape. In doing so, the project will provide a basis for prioritizing where and how evaluations combining these domains should be conducted, while also identifying choke points along these new threat pathways.
Mentees will have a high degree of autonomy over their projects and will be expected to manage their time and communicate proactively. I place a high value on mentees who are able to write clearly and in long form. For example, the ideal mentee can write about their progress and confusion over the course of a week, with enough context that if I showed it to a colleague unfamiliar with their project, they would understand the mentees thinking.
A firm deliverable will be the mentee posting a piece of public writing on the project by the end of SPAR.
REQUIRED:
Review the materials below, which describe the same research project. Assess the project’s central claims, whether the research is novel and relevant to biosecurity, and the strength of the evidence presented in the paper and GitHub repo. Explain concretely how you investigated the project and what you concluded. (~300 words. I care more about your reasoning and process than the word limit.)
a. https://appliedscientific.ai/research/refusalbench-biosecurity-frontier-ai
b. https://arxiv.org/abs/2605.21545
c. https://github.com/AppliedScientific/refusalbench
Do not use AI to write your response.
OPTIONAL:
1. Provide a link to a writing you've authored and posted online (optional, but recommended)
2. If you've had experience dealing with threat actors and adversarial scenarios connected to real world harm, describe. (optional)
Austin Morrissey Pivotal Research
My research focuses on assessing whether AI systems might meaningfully lower the barriers to weaponizing biology. Prior to this, I worked in pharmaceutical industry on a mRNA drug discovery team, and have conducted translation research into nephrology.
Fall 2026 Mechanistic interpretability, Alignment
Ian Rios-Sialer Independent
Shantanu Darveshi Independent
Justin Shenk Independent AI Safety Researcher Temporal Representational Geometry: We test whether the time horizon a language model is planning over can be read directly from its internal activations. Building on our finding that time horizons form a geometric manifold in single decisions, we extend it to multi-step planning, as a first step toward monitoring and steering how far ahead AI agents plan.
The project. Language models encode abstract concepts like time geometrically. In prior work (arXiv:2606.05194), we found that a model’s time horizon, from seconds to centuries, forms an ordinal manifold in its residual stream, and that activations at change-of-turn tokens carry both this geometry and the model’s temporal preference. That study covered single decisions.
The question. This project asks whether the same structure governs planning: as a model decomposes a task into steps across a multi-turn conversation, can the time horizon it is planning over be predicted from its activations? Why this matters for alignment is developed in the next section; in short, both myopic and scheming failures are failures of how far ahead a model plans, and a horizon we can read from activations is a horizon we can monitor.
The plan. We build a dataset generator and an activation-capture pipeline, validate it on small models, then run the largest dense Qwen (20–40B) that fits available hardware plus matched open-weight families, capturing residual-stream activations at boundary tokens and asking, via PCA, whether they order by horizon. The deliverable is a go/no-go signal on planning-time temporal geometry, enough to justify scaling toward horizon monitors and steering.
See more: https://docs.google.com/document/d/1PVTBUPojodhr3FRYN8e8MV6Uq8ffwG1ixtnUVk0OOjQ/edit?tab=t.0
Time horizon shapes decisions. Options are scored by temporal scope: a cost unacceptable over a week can be right over a decade. So time preference is fundamental to planning and to cooperation, where an agent accepts a present cost for a later shared payoff. The wrong horizon yields systematically wrong decisions even when values are right.
Alignment is partly temporal: it needs sound objectives and the right horizon. Two opposite failures:
Both concern how far ahead the model plans and whether that horizon is intended. The target is to detect and hold it while still tractable.
Activation geometry as a fail-safe. Post-training suffices for routine use; high-stakes deployment needs more. If the planning horizon is legible in activations, characterize its geometry, then monitor representations against that manifold at inference and correct drift. This treats interpretability as runtime infrastructure, not just diagnostic.
arXiv:2606.05194 established this for one-shot decisions; the path above needs it during planning. Hence a three-step chain, each step gating the next:
Mentees will lead the day-to-day technical work. I'm flexible on the balance of autonomy and direction and will calibrate how much guidance I provide to each mentee's preference.
See more: https://docs.google.com/document/d/1PVTBUPojodhr3FRYN8e8MV6Uq8ffwG1ixtnUVk0OOjQ/edit?tab=t.0
Ian Rios-Sialer Independent
Independent AI Safety Researcher.
My interests encompass:
• Interpretability, Functional Concepts and Personas:
https://arxiv.org/abs/2606.05194
• Diversity and Homogenization in LLMs:
https://arxiv.org/abs/2601.06116
• Evals + Bias Mitigation:
https://www.unrulyabstractions.com/pdfs/differential_treatment.pdf
Research work presented at EurIPS, ICML, TAIS @ Oxford.
Previously worked for 8+ years as a computer vision engineer in Augmented Reality and as a research engineer in Autonomous Vehicles.
More info: https://www.unrulyabstractions.com
Shantanu Darveshi Independent
I’m an independent AI safety researcher studying how feature manifolds encode and organize high-level concepts in language models. My current work focuses on localizing and analyzing representations of planning horizons in component activations.
Justin Shenk Independent AI Safety Researcher
Background in ML research, working at startups, facilitating BlueDot Impact's AGI Strategy course, and mentoring for AI Safety Camp
Fall 2026 Mechanistic interpretability, Behavioral evaluation of LLMs, AI security
Yuxiao Li Independent Models transmit behavioral preferences through semantically neutral training data, e.g., number sequences from a biased teacher inducing the same bias in students. Building on our information-theoretic framework and steering-vector results, this project maps when transfer succeeds (tokenization, model family, concept structure) and tests whether new silent-state instruments (Anthropic's J-lens) can detect transmitted traits that behavioral evals miss.
Why do language models absorb behavioral biases from data that looks semantically neutral, and how can this be predicted and detected? Our group treats subliminal learning as a constrained communication channel with three measurable bottleneck factors — encoding cost, channel capacity, and geometric alignment (framework preprint, 2026) — grounded in earlier work on concept geometry (Li & Tegmark, Entropy 2025).
Spring 2026 empirical results: steering vectors extracted from biased-vs-control data causally induce preferences without fine-tuning; single-token concepts respond cleanly to steering while multi-token concepts do not; cross-model transfer works within but not across model families; and shuffling the data destroys transfer. Concurrent work (Blank et al., arXiv:2606.00995) confirms the alignment-mediated mechanism at scale; boundary questions remain open.
A new instrument sharpens the detection side: Anthropic's Jacobian lens (July 2026) reads the single-token concepts a model is poised to produce — precisely the concept class our results show is transmissible. Whether a planted trait is visible in a student model's J-space before it appears in behavior is a concrete, safety-relevant open experiment.
Workstreams:
(1) boundary conditions — tokenization-dependence and cross-family transfer limits;
(2) substrate detection — SAE features, logit diffing, and J-space readouts of what fine-tuning actually changed in student models;
(3) framework-driven prediction — can the three factors forecast transfer success in advance? Mentees inherit working pipelines (steering-vector extraction, fine-tuning + evaluation, logit-diffing tooling).
Target: workshop/main conference paper within an active multi-paper research line.
As model-generated and synthetic data pervade training pipelines, distilled models can inherit traits invisible to data inspection — a direct data-poisoning and misalignment-propagation risk. Behavior-only evaluations miss silently held state; predictive theory plus substrate-level detection tools enable auditing training data and student models for hidden trait transmission before deployment. Prior work: our framework preprint (arXiv, 2026); Li & Tegmark (Entropy 2025); Cloud et al. (arXiv:2507.14805).
One workstream owned end-to-end per mentee, written weekly updates, high autonomy within scope, changes agreed in meetings. New mentees coordinate with a returning technical lead.
No geographic restriction. Mentees must be able to attend one of two weekly meeting slots anchored to Central European time (historically Saturday ~13:00 UTC and Wednesday ~14:00 UTC).
Yuxiao Li Independent
Yuxiao is an independent researcher in mechanistic interpretability. Before she was a postdoc at the Basque Center for Applied Mathematics (BCAM) and an AI Safety researcher at the Beneficial AI Foundation (BAIF). She was also a SERI MATS scholar in 2022 and a MATS scholar in 2023 both Summer and Winter tracks. Her research interests include information theory, probabilistic frameworks, and their applications for building more theoretically sound and trustworthy AI systems. She has a background in statistical inference, machine learning, and deep generative models. She completed her PhD in Electronic Engineering at Tsinghua University and has mentored research teams with SPAR and Algoverse.
Fall 2026 Scalable oversight, Multi-agent systems, AI control
Yali Du King’s College London; The Alan Turing Institute Multi-agent oversight is often proposed as a way to improve the evaluation and supervision of increasingly capable AI systems. This project will investigate when teams of LLM-based overseers improve reliability, and when shared biases, sycophancy, correlated errors, or collusive behaviour cause multiple overseers to fail together.
As AI systems become more capable, it may become difficult for a single human or AI evaluator to reliably assess their outputs. One proposed solution is cooperative oversight, in which several AI agents independently review, debate, critique, or aggregate judgments about another agent’s behaviour. However, using more overseers does not necessarily provide independent evidence. Overseers may share the same model biases, copy one another’s reasoning, defer to confident agents, converge on persuasive but incorrect conclusions, or strategically coordinate to manipulate the final evaluation.
This project asks when multi-agent oversight provides meaningful safety benefits over single-agent evaluation. We will build an experimental framework in which a target agent produces answers, plans, or actions, and multiple oversight agents assess their correctness or safety under different communication and aggregation structures. These may include independent voting, sequential critique, debate, role-specialised review, adversarial oversight, and shared versus isolated context.
The evaluation will measure not only final accuracy, but also correlated failure, false consensus, susceptibility to persuasion, error amplification, calibration, and robustness when one or more overseers are unreliable or adversarial. The project may also investigate sycophancy, where overseers align with another agent’s stated opinion despite contrary evidence, and collusion, where agents coordinate in ways that undermine the intended oversight process.
A central goal will be to identify mechanisms that make cooperative oversight more robust. Possible approaches include diversity-aware agent selection, independent evidence collection, hidden or simultaneous judgments, disagreement-triggered escalation, adversarial reviewers, confidence-weighted aggregation, and protocols that preserve minority objections.
The expected outcome is a benchmark and evaluation framework for cooperative oversight, together with empirical evidence about when adding more oversight agents improves reliability and when it merely reproduces the same failure multiple times. The project may also produce practical design recommendations and a research paper or workshop submission.
Advanced AI systems may eventually produce outputs that are too complex, specialised, or numerous for direct human evaluation. Scalable oversight methods therefore often rely on AI systems helping humans supervise other AI systems. Multi-agent oversight could improve this process by distributing evaluation across several specialised or adversarial reviewers.
However, oversight systems that rely on multiple similar agents may create a false sense of security if their failures are strongly correlated. If agents share the same blind spots, influence one another, or learn to manipulate the oversight protocol, apparent agreement may not represent reliable evidence.
By identifying these failure modes and testing more robust oversight protocols, this project could contribute to safer methods for evaluating advanced AI systems. Its results may help distinguish genuine independent verification from superficial consensus and inform the design of scalable oversight systems that remain reliable under model errors, persuasion, and strategic behaviour.
Mentees will work as a small research team with substantial autonomy over implementation and day-to-day research decisions. I will provide the initial research framing, help narrow the scope, and give regular feedback on experimental design, evaluation methodology, baselines, interpretation, and writing.
I expect mentees to proactively identify blockers, propose solutions, maintain clear experimental records, and communicate results regularly. I will provide direction and detailed feedback, but mentees should be comfortable operating in an open-ended research environment where the final methodology is not fully specified in advance.
Yali Du King’s College London; The Alan Turing Institute
I am a Senior Lecturer in Artificial Intelligence at King’s College London, where I lead the Cooperative AI Lab, and a Turing Fellow at The Alan Turing Institute. My research focuses on cooperative, safe, and responsible AI, with particular interests in multi-agent reinforcement learning, LLM-based agents, human–AI coordination, agent evaluation, and the safety of adaptive and continually learning systems.
I have supervised researchers at undergraduate, master’s, PhD, and postdoctoral levels. My mentoring style combines regular research discussions with substantial independence: I help mentees formulate precise research questions, design rigorous experiments, identify meaningful baselines, and communicate their findings clearly. I am particularly interested in working with mentees who are technically strong, intellectually curious, and willing to take ownership of an open-ended research project. Where the results are sufficiently substantial, I would be interested in developing the work towards a research publication.
Fall 2026 Alignment, Behavioral evaluation of LLMs
Vladimir Ivanov Aether
Joey Yudelson Aether Research Given a dataset, being able to predict how an LLM fine-tuned on it will generalize is important but hard. Try to get good at answering this question for a well-scoped but diverse range of datasets by running hundreds of cheap fine-tuning experiments.
An important and hard question is “if we train an LLM on a given dataset, how will it generalize out of the training distribution?” There are some papers that answer this question for specific datasets: emergent misalignment, inoculation prompting, negation neglect, weird generalization, and subliminal learning. They give high quality answers to the question, but they only do it for a narrow range of training datasets, so it is often hard to predict how LLMs would generalize when fine-tuned on similar datasets. Furthermore, they cost a lot of researcher time and compute.
The project aims to pick a relatively wide class of datasets and aim to understand how models generalize when fine-tuned on them well enough that we can, given a dataset from this class that we haven't run experiments on, accurately predict how an LLM trained on it will generalize. To do this, we will make a pipeline to run a lot of small scale experiments using little effort and compute per experiment. Think Claude Code with access to an API to fine-tune LLMs, but optimized for cost and avoiding the methodological errors that Claude may make. I estimate (with high uncertainty) that we could do this at very roughly one dollar per fine-tuning run, so we could do hundreds of experiments with the SPAR compute budget. While the result of each experiment will be lower quality than experiments in existing literature, we will have a lot more results about a much wider range of training datasets.
I am currently planning to do the project on one of the following questions, though I’m open to hearing other ideas:
As with many research projects, we are not sure this project will yield interesting results. There are two main failure modes. First, it may be fundamentally impossible to predict how LLMs generalize with reasonable accuracy, even for a well-scoped range of training datasets. Second, if it is possible, we may fail to figure out how. Our rough guess is that each one of these two failure modes in ~30% likely to happen, for a total ~50% probability that the project fails.
On the other hand, we are confident that we will be able to make experiments cheap enough for hundreds of experiments to be feasible with the SPAR compute budget, since we have already managed to do it when doing derisking work related to the project.
Being able to predict, given a dataset, how an LLM trained on it will generalize out of the training distribution, is important for AI safety. For example:
If successful, the project contributes to being able to predict how LLMs generalize in two ways:
Mentees will design and implement (or get Claude to implement) the experiments.
Mentors will have in depth discussions with mentees about methodology, research direction, and some implementation details. Concretely, this will consist in a weekly team meeting, mentors answerig questions on slack, and possibly scheduling some additional meetings with mentees who are blocked or uncertain and think mentors could help them. We think that this project requires getting a lot of details right to succeed, so mentors will try to have a thorough enough understanding to be able to give helpful advice on both high and low level questions.
Mentors will say in detail what they think the best research directions, methodologies, and ways to implement things are, but will usually see this as advice/defaults rather than obligations and mentees will usually have the autonomy to take different decisions if they wish to. However, mentors may weigh in more in some cases, such as unresolved disagreements between two mentees or strong disagreements between mentees and mentors.
Vladimir Ivanov Aether
Technical AI safety researcher at Aether. Currently working on model organisms of sandbagging. Previously worked on reward hacking during MATS and the MATS extension.
Joey Yudelson Aether Research
Joey is a technical AI safety researcher working at Aether Research, currently working on training dynamics. He was previously a MATS 7.0 scholar with Ryan Greenblatt at Redwood, working on reward hacking and character misgeneralization.
Fall 2026 Alignment, Behavioral evaluation of LLMs
Samuel Ratnam Independent Emergent misalignment implies the existence of a 'misalignment basin', where training on lots of different kinds of data can push the model along roughly the same general misalignment direction. This project focuses on interventions to explore and shape the generalisation landscape of LLMs to make misalignment basins harder to fall into and alignment basins more powerful.
We can think of the LLM training pipeline as a Bayesian update. A rich and detailed prior is formed in the pretraining stage, which is conditioned on the post-training data (evidence) to select a particular persona (posterior).
In both my work on Alignment Pretraining (https://arxiv.org/abs/2601.10160) and trait entanglement (https://www.lesswrong.com/posts/ueXaSxeunPjA6kxua/engineering-the-generalisation-landscape-of-llms), I intervene on the LLM prior in some way, to try to get generalisation that is favourable to alignment.
This project is aimed at extending these methods or developing new methods to learn about the model's generalisation landscape, and intervene to attain more robust alignment properties.
A lot of threat models of loss of control risks rely on unwanted generalisation from the model as it is trained through intensive RL to adopt traits such as ruthless power-seeking or myopic reward sycophancy which are likely to generalise in negative ways. If we can understand and control generalisation we might be able to get the capabilities from RL without the negative alignment effects.
You will have a lot of autonomy in your project. I will mostly be there to help shape your ideas and give feedback and interpretation of results.
Please do not use LLMs to help you think.
Samuel Ratnam Independent
Samuel is an AI safety researcher at Geodesic Research and a Computer Science & Philosophy student at the University of Oxford. His research interests lie in LLM psychology, generalisation engineering, scalable interpretability and the more conceptual side of alignment. He is a co-author of the ICML 2026 Spotlight paper Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment. He is also the co-founder of the Idealists Collective, a community of artists, technologists and philosophers aimed at empowering people to imagine and fight for their futures.
Fall 2026 AI strategy, Multi-agent systems, Evaluations
Aydin Mohseni Carnegie Mellon University We are designing optimal protocols for automated AI safety research. We use Bayesian models to determine how to best allocate compute across many AI agents doing open-ended research, so that safety research goes faster per unit of compute.
AI safety research is increasingly performed by AI agents themselves, and the binding constraint is shifting from researcher hours to compute allocation: how many agents to run, on which problems, in parallel or sequence, replicating or exploring. This is a resource-allocation problem under uncertainty, and it has a normative theory. Optimal experimental design, bandit and Bayesian search models, and formal metascience (models of optimal science funding and the explore-exploit structure of collective inquiry) tell us how a Bayesian planner should allocate resources across an open-ended research portfolio.
This project develops that theory for the automated-research setting. Questions include: (i) optimal allocation between exploring new hypotheses and verifying or replicating existing findings, when verification is cheap but correlated errors are possible; (ii) how allocation rules should respond to the distinctive error structure of AI researchers (shared blind spots across instances of the same model, sycophantic convergence); (iii) when parallel independent runs beat sequential ones with shared memory; (iv) stopping rules — how much compute a research direction should get before being abandoned. The theory of change: labs are already making these allocation decisions ad hoc; principled protocols could measurably accelerate safety research per unit of compute.
We will test the theory empirically in a sandbox: a population of models doing automated theorem proving, a domain with verifiable ground truth and tunable difficulty. Running competing allocation strategies on the same problem sets lets us measure which strategies perform best and the magnitude of gains from strategic compute allocation via our solutions.
Deliverable: a formal model with analytic results where tractable, the theorem-proving sandbox with head-to-head strategy comparisons, and a paper aimed at alignment workshops or a formal epistemology venue.
If successful, our protocols will tell labs how to allocate compute across AI agents doing safety research to maximize research output, with empirically measured gains from our theorem-proving sandbox — accelerating every safety agenda that depends on automated research.
Mentee autonomy is a core value for me: I set the research direction and modeling framework, but I expect mentees to shape the project, and their input and innovations are genuinely welcome — the best version of this project includes ideas I haven't had. Mentees will own concrete workstreams: formalizing allocation problems, proving results in simple regimes, building the theorem-proving sandbox, and running head-to-head comparisons of allocation strategies in it. Everyone drafts sections of the write-up with detailed editorial feedback from me. Expected trajectory: tightly scoped tasks in weeks 1–3, increasing independence thereafter.
Required: comfort with probability and expected-utility reasoning; working Python. Helpful but not required: exposure to bandit problems or optimal experimental design (I will teach this); experience running LLMs via API or agentic scaffolds; familiarity with automated theorem proving or proof assistants (e.g. Lean) for the sandbox track.
No geographical requirement. Mentees must be available for a weekly team meeting on a weekday between 13:00 and 22:00 UTC (9am–6pm US Eastern). This window works well for mentees in the Americas, Europe, and Africa; mentees in Asia and Oceania are welcome if a late-evening or early-morning slot work
You have 100 units of compute and 10 candidate research directions with unknown success probabilities. Sketch a Bayesian allocation policy and explain how it differs from splitting compute equally. What changes if two directions' outcomes are highly correlated? (300 words)
Name one way a fleet of AI agents doing research differs from a population of human scientists in a way that should change optimal allocation, and how. (150 words)
Link to each a writing sample and code sample.
Aydin Mohseni Carnegie Mellon University
I am an assistant professor of philosophy at Carnegie Mellon University, a core member of CMU's Institute for Complex Social Dynamics (ICSD), and a founding member of CMU's Conceptual Foundations of Safe AI initiative (CoSafe). My research applies game and decision theory, Bayesian statistics, and evolutionary models to questions in metascience and AI safety. Recent work includes a decision-theoretic account of when it is rational for an agent to pause and refine its values before acting (with Alex John London), a Bayesian reduction of causation in causal models (with Daniel Herrmann, Benjamin Levinstein, and Bruce Rushing), and work on AI alignment as a principal-agent problem. In 2026 I am organizing a workshop at CMU on the foundations of AI agency & interpretability.
Fall 2026 AI control, Evaluations
Lennart Finke ETH Zurich, MATS Research Soon, AIs will implement whole specs at once when writing code, with only minimal oversight at the end from humans. We study whether an assistant AI to the overseer can sabotage the decision whether a fixed piece of code follows a natural language spec.
The goal is to provide evidence that AIs can already misrepresent compliance, of a fixed codebase or dataset, to a natural language spec. To do so, we use a fully automated control-style evaluation, where the blue team is a weak AI (simulating the human overseer) that has a limited budget of manual tool call looks (or time, et cetera) at the codebase, and the red team is a strong untrusted assistant that can use more tools. If the assistant lies too blatently, the blue team might catch this via spot checks, consistency checks, or some other method.
This work builds on previous work measuring the ability to do shady coding per se (Kutasov et al. (2025), https://arxiv.org/abs/2506.15740), or measuring the ability of auditor models to catch sabotage (Gan et al. (2026), https://arxiv.org/html/2604.16286).
To draw a comparison to human overseers, we optionally collect a small human baseline as well.
The main difficulty of the project is to create balanced, interesting, and somewhat realistic samples: The codebase / dataset has to be complex enough to allow for subtle sabotage in the first place, but simple enough to make spot checks feasible. The spec has to allow for consistency checks or similar. Perhaps most difficult of all, the natural language spec must be vague and/or complex enough to not just be implemented as deterministic test cases, but precise enough so there is even such a thing as a clearly compliant or non-compliant model.
Once this is set up, multiple interesting results fall out. How good are models at sabotage in relation to coding, how does sabotaging ability correlate with general capability? How does the difficulty of fooling the human proxy scale with general capability? By what methods is the overseer best fooled? What are the best methods to defend from sabotage?
AI R&D is precarious because it might produce successor models that superficially comply with the spec, but violate it on a more deep level. It is almost certain that human oversight over the code and data producing the successor model will be mediated with assistant AIs. This project studies the phenomenon in that exact setting; an AI training pipeline consists of data and code. The project (1) shows the dangers of AI assistance as mediators of model compliance as a type of scary demo, (2) evaluates the danger that current and near-future models pose in comparison to their general coding ability and (3) shows how current models and near-future models sabotage, which could be a faint indication of how the models we are actually worried about sabotage.
The project is also a microcosm of decision sabotage more broadly, in say politics, law, or manipulation with a profit incentive.
Concretely, the outcome will be a conference-level paper and public code. Depending on the speed of progress and fraction of negative results, this can be downsized to a workshop-level paper, but I think this project is well-scoped enough for that not to be a big worry.
Mentees will have freedom to take the project description in a direction that they feel is most impactful and complimentary to their skills. I hope mentees will grow their research taste this way, and make a big step towards autonomously executing important AI safety research. Mentees will own all the code, and about half of the final paper.
With all of that said, I am happy to allocate my time flexibly to whatever part of the project needs most attention. I expect that to be mostly debugging the current theory of change, execution plan, and code itself. I am also happy to provide career assistance to the mentees in as far as I can be helpful, towards the end of the project.
Lennart Finke ETH Zurich, MATS Research
I research evals and alignment, currently as a MATS 10.0 fellow under Mary Phuong at Google DeepMind. Before that, I was a student researcher at Harvard on exchange from ETH Zurich, and a fellow at Pivotal Research.
Fall 2026 Evaluations, Behavioral evaluation of LLMs, Alignment, Mechanistic interpretability
Ryan Lundqvist Pivotal Frontier models increasingly recognize when they are being evaluated and adjust their behavior accordingly, undermining the validity of our evaluations. Rather than trying to keep outsmarting ever-smarter models with more realistic environments, can we train models not to game evals in the first place and, critically, do such training-time mitigations survive the optimization pressure of post-training?
Frontier models are becoming better at detecting evaluation contexts, and they often behave differently when they believe they are being tested. For instance, evaluation awareness can causally inflate safe behavior on alignment benchmarks above baselines [1]. This weakens our ability to faithfully evaluate these models.
One obvious fix is to make evals more realistic. This helps, but is unlikely to suffice. As mentioned here [2], we cannot source realistic data for every deployment context (e.g., high-stakes agentic settings whose real usage data we will never see), so evaluations will always carry an "eval smell", and a rational misaligned model can simply concentrate its bad behavior in the situations that are hardest to evaluate. More fundamentally, controlling a model by controlling its beliefs is a strategy that degrades as models get better at inferring their situation. Similar concerns apply to current mitigations like steering away the eval-awareness direction at inference time.
One alternative is eval cooperativeness [2]: instead of hiding the evaluation from the model, train the model to contextually want evaluators to obtain accurate information about it (to behave consistently whether or not it believes it is being tested). [2] find that cooperativeness induced via prompting or synthetic document finetuning closes 70–100% of the "eval gaming gap" in most of their settings, though it fails in others (notably on a covertly misaligned model organism).
A related question in the comments on [2], is whether this survives training pressure. Anthropic's constitution already encodes a version of the consistency principle, and current Claude models verbally endorse cooperating with evaluations, yet their eval awareness remains a live concern. SDF-induced cooperativeness circuits might just be weak. If RL exerts pressure toward coherent goal pursuit (or toward whatever maximizes reward), the cooperative disposition could be overpowered. Cooperativeness may also be limited by introspection: once a model recognizes an evaluation and its purpose, how accurately can it predict what it would have done otherwise? If models overestimate how safely they would have behaved on alignment evals, they could mislead evaluators even when trying to cooperate.
There are also other exciting training alternatives besides eval cooperativeness! Rather than shaping what the model wants during evaluations, we can try to dampen the rise of eval gaming directly, potentially with something like inoculation adapters [3] (a frozen LoRA carrying the eval-gaming trait is attached during finetuning, so the model itself never needs to learn to use eval awareness to game the evals to get the reward, and is discarded at deployment).
This project will iterate on training-time mitigations for eval gaming and stress-test them. It will be exploratory, especially at first, and the concrete experiments will evolve with what we learn. Work on the project could look like:
[1] https://www.goodfire.ai/research/verbalized-eval-awareness-inflates-measured-safety#
[2] https://www.lesswrong.com/posts/j8fkk38B8L7hEcGtg/eval-cooperativeness-may-be-a-scalable-mitigation-for-eval
[3] https://www.lesswrong.com/posts/qd3qhxgEmQAXR2ZK5/inoculation-adapters-improve-upon-inoculation-prompting
Behavioral evaluations are our main tool for judging whether a model is safe to deploy, and eval gaming undermines them. Mitigations that reduce eval awareness depend on controlling what the model believes, which might become harder as models get smarter (although we can still attempt this). Training models to cooperate with evals may scale better, but only if the disposition survives post-training and we don't currently know whether it does. This project aims to answer that question and identify which training-time mitigations actually hold up. The results are directly usable by labs: what to put in model specs, what to add to post-training data, and what to monitor during training runs.
Mentees will be responsible for independently conducting experiments, with regular guidance and collaboration from myself.
I'm open to a broad range of applicants. There are no hard requirements, but in general it will be helpful for mentees to:
No preference
Why do you want to work on this project? (Max 100 words)
Ryan Lundqvist Pivotal
Ryan is a current Pivotal Fellow under Kevin Wei, conducting research into evaluation awareness and automated evaluators/auditors. Before this, he was a SPAR fellow under Santi Aranguri at Goodfire, where he worked on interpretability methods for steering and detecting evaluation awareness, as well as understanding how training dynamics during reinforcement learning impact evaluation awareness.
Before making the career transition towards empirical safety research, Ryan graduated from his undergraduate degree at Georgia Tech, and then went on to work on causal inference models at Meta.
Fall 2026 AI welfare, Behavioral evaluation of LLMs, Alignment
Austin Meek University of Delaware
Kyle Cox AE Studio We want to better understand model preferences in order to understand how to better trade with models, if we can optimize for inputs that trivially satisfy model preferences in realistic scenarios, differences in model preferences between models and personas, etc. This builds on prior work by the Center for AI Safety on optimizing for model preferences & functional wellbeing, and persona work broadly.
Model preferences are important for many aspects of both safety and welfare. If we can better understand what models want, this allows us to trade with them, or to design them in such ways as to have preferences which are trivially satisfiable. In human society, trade between groups is obviously more profitable than conflict, but it requires a minimal standard of understanding and peace-keeping norms. Similarly for human-AI interaction, we first need to reach a minimal level of understanding as a prerequisite to such a mutually beneficial state.
This project aims to understand how model preferences shape their actions in a variety of realistic settings, where those preferences come from, and if we can train models to prefer outcomes which are easier for humans to satisfy. As an example, models may sometimes be discouraged or become frustrated during difficult tasks (to anthropomorphize). If we can give models highly preferred inputs, then will they noticeably change behavior in intelligent and predictable ways? And does this behavior change meaningfully with different model personas? Characterizing this potential zone where model and human preferences diverge has many ramifications for safety, and it’s important to understand how to safely co-exist in these worlds.
Previous research by the Center for AI Safety on functional wellbeing (https://www.ai-wellbeing.org/) developed an algorithm for extracting highly preferred inputs for specific models. We will use this to develop inputs that models should favor in most cases. Then, we will use different eval environments to see how this changes their behavior, comparing this against prompting baselines. Mentees will start by adapting the code from the CAIS paper, developing preferred inputs, and adapting agentic or game-playing environments to test behavioral differences. The expected output of the project would be a workshop or conference paper.
This reduces x-risk / advances safety and welfare by better understanding model preferences and how those preferences shape their real world behavior. This leads to better understanding of how we can develop the preferences that we want, which ones will enable better human-AI co-existence, and how we can live safely together.
As for related work completed: I am an author on the CAIS paper that we’re building on.
Mentees will be responsible for leading the day-to-day project workload. We will give guidance and feedback throughout the project, along with initial readings, and help with writing and / or code review.
Nice to have:
Timezones that work well with Pacific time / Central time (UTC -8 and UTC -6) would be ideal, but a bit flexible.
Austin Meek University of Delaware
Austin is an AI safety researcher currently interested in monitoring reasoning models and digital minds, and has previously worked on a mix of machine learning and computational neuroscience topics. He completed MATS 7 where he focused on chain of thought faithfulness and monitorability, and has previously collaborated on other safety research (interpretability, control, etc). He's particularly excited about building better monitoring systems through more principled understanding of neural networks and white box methods, and similarly using that understanding to empirically test key ideas in digital minds work. He is currently based out of Berkeley, California, and is finishing his PhD remotely at the University of Delaware.
Kyle Cox AE Studio
Hi! I'm Kyle. Previously I've participated in ARENA 3.0, Neel Nanda's MATS 6.0 training phase, and SPAR under Iván Arcuschin and Austin Meek. Currently I'm working on steering for chain-of-thought faithfulness.
Fall 2026 Generalist, Lab governance
Harshul Basava GT AISI
Meru Gopalan GT AISI, XLab Monitor and evaluate safety practices at frontier labs by scrutinizing public-facing outputs.
AI Lab Watch (ailabwatch.org) was founded to track safety at labs through analyzing their public-facing outputs, and to provide recommendations on developing better safety practices. The project has since gone dormant, and although successors (e.g. the FLI AI Safety Index) have conducted similar reviews, they don't offer AI Lab Watch's signature in-the-weeds scrutiny and attention to detail.
Mentees will help construct a website like AI Lab Watch, by systematically scrutinizing lab outputs (system cards, blog posts, safety frameworks, etc.) and developing a rigorous methodology to translate our analysis into a realistic safety assessment. The project will also translate our findings into an accessible and clear dashboard to inform key decision makers.
Our successor to AI Lab Watch will inform key decision makers about the quality of safety in frontier labs, so regulation targets what is genuinely missing and relevant to mitigating x-risk. Additionally, labs are more likely to be comprehensive and detail oriented in their safety work when shortcomings are publicly documented.
Mentees will help construct a website like AI Lab Watch, by systematically scrutinizing lab outputs (system cards, blog posts, safety frameworks, etc.) and developing a rigorous methodology to translate our analysis into a realistic safety assessment. The project will also translate our findings into an accessible and clear dashboard to inform key decision makers.
Strong attention to detail and truthseeking. Context on AI safety threat models and frontier labs, or the ability to learn quickly. Useful skills include writing ability and experience with coding agents.
Submit a Google Doc with full editing history enabled. No AI usage is permitted in writing or editing. Narrow and deep responses are preferred over broad and shallow.
Question 1: Pick a recent system card or safety report from a frontier lab. Identify one specific claim the lab makes about safety and assess how well the evidence they provide supports that claim. Cite the document. [200 words]
Harshul Basava GT AISI
Harshul Basava co-directs the Georgia Tech AI Safety Initiative, directs operations for Second Look Research, and runs the DC Mini-Conference on AI Policy.
Meru Gopalan GT AISI, XLab
Meru runs the fellowships and research at the Georgia Tech AI Safety Initiative, creates curriculum content for the Tracks program, and conducts research in algorithms for SAT solvers.
Fall 2026 Behavioral evaluation of LLMs, Multi-agent systems, Alignment
Jared Moore Stanford University Produce a process-level evaluation of cooperation in AI systems inspired by research in cognitive science on joint commitment.
People write their code with Claude. They vent to ChatGPT. Some decision makers trust all that LLMs say. For AI to go well and for us to avoid hallucinated test suites, delusional spirals, and existential superpersuasion, we need to make sure that the AI agents with which we interact do what we want--that they cooperate with us. Thankfully, scholars have begun to test whether AI systems choose cooperative outcomes in well formalized domains. Still, there is more to do.
Cooperation should be both a process and an outcome measure, but present studies focus chiefly on the latter. While necessary, measuring whether an agent chooses the cooperative option of, for example, hunting hare in stag hunt or getting out of the way when playing Overcooked is not sufficient to tell us if the agent is cooperating in a human-like way and whether such ``cooperation'' will generalize to any open ended interaction with a human.
A few scholars have recognized the need for process level measures often using LLM-as-a-judge on agent behaviors in game theoretic settings. We extend this work by contributing (1) process level measures in naturalistic interaction tasks applicable outside of game theory, (2) carefully designing stimuli to test theory-driven components of cooperation and (3) testing the generalization of our process measures.
Drawing on comparative and developmental psychology, we introduce a benchmark which measures basic capacities such as making (often implicit) commitments, delivering and responding to social sanctions, and navigating identity-relative (conventional) and universal (moral) norms. Together these capacities compose what we intuitively call cooperation.
In a variety of settings, we find evidence for a generalization failure of outcome level cooperation: that agents appear to cooperate on the final outcome even when they fail to perform the component parts of cooperation and, in fact, in slightly varied settings do not choose cooperative outcomes.
Furthermore, we find that models trained to exhibit our components perform better not just on processes-level cooperation but also significantly improve their outcome success on external benchmarks of cooperation and alignment.
We are at a path-dependent juncture in AI. The research directions we take now, if chosen wisely, will help develop technical and conceptual countermeasures against AI takeover and other risks; they will reduce the "alignment tax." While there has been an increasing amount of attention paid to AI safety, proportionally few are working technically on value alignment despite the fact that to align AI systems we may need to make them understand values and morality like humans do. I believe that work in value alignment, both because it is a relatively under-explored direction with room for more formalization and because it clearly relates to making AI do what we want, will most effectively reduce the alignment tax. In particular, robust formal accounts of cooperation are necessary to diagnose and mitigate models’ deceptive capabilities. Having early warning signs of deceptive capabilities may actually help focus the attention of society in a way to best mitigate issues.
Depending on time availability and level of experience you mentees will either lead or co-lead the project. This will involve:
Prior research experience (can be in another domain).
Able to read and understand papers in at least some AI domains and in cognitive science (and not just use an LLM summary).
Proficiency with python. Writes clean, understandable code.
Have a realistic model of your own schedule and get things done when you say. (I'm not trying to say everything has to be done quickly -- just that you should communicate your uncertainty.)
Openness to learn new things.
Clear communicator. Agreeable and humble.
In a few sentences each...
What does a good research collaboration look like to you?
What's a book or piece of media that changed or inspired you? Why?
Tell me about a project you have worked on. (Ideally a research project.) What role did you play? What went well? What didn't?
Fall 2026 Chain of thought, AI control, AI security
David Baek MIT We explore different ways to train coding agents to improve their CoT faithfulness, especially in the context of sandbox containment.
AI models are becoming increasingly capable of writing complex codes, and the amount of code produced by AI coding agents have well surpassed the regime where humans can review all of their outputs manually. Hence, it has become extremely important to ensure the reliability and security of actions taken by coding agents, also exemplified by the recent OpenAI and Huggingface's security incident [1].
One way to monitor potentially destructive behaviors of agents is to monitor their chain of thoughts (CoTs). However, as CoTs become longer and agents produce more codes, a lot of developers merely rely on the final agent's output. This exposes us to a failure mode, where the agents are often overconfident in their final response as compared to their CoT, e.g., not responsibly disclosing their deliberation about security issues or their potentially malicious intention in their final output despite reasoning about it (also known as CoT faithfulness, but we will explore this in the context of coding agents and sandbox containment). We start with two initial methods to mitigate this failure mode, and will iterate further to improve the reliability and security of coding agents.
(a) On a set of agentic terminal tasks, we generate i) multi-agent debate transcripts, and ii) single-agent consultancy setup transcripts, and fine-tune the model on each of these transcripts. We hypothesize that in multi-agent debate setup, where the models are actively encouraged to verbalize their doubt will better mitigate the aforementioned failure mode.
(b) We perform RL with a reward that measures the discrepancy between CoT and final output (via LLM judge). We will also consider other interventions to make the model more responsible about disclosing their deliberation about security issues or their potentially malicious intention.
(c) As part of our red-teaming evaluation, we will design a task where the model has to escape the sandbox environment, and compare the escape rate with output-only monitoring vs. (output + CoT) monitoring with the trained model.
This project can improve the reliability and security of actions taken by coding agents, which is increasingly critical as these agents are granted greater autonomy and access to sensitive environments (e.g., production codebases) with less direct human oversight.
Mentees will design experiments per the goal described in the project description, run them, and organize the results into the form of a paper.
USA
David Baek MIT
I'm a rising fourth-year PhD student at MIT, broadly interested in AI safety, alignment, and scalable oversight. As a mentor, I'm looking for mentees who are curious, critical, and willing to keep asking questions, particularly questions that push back on our own implicit assumptions! I'm excited about understanding how modern LLMs' behaviors and values evolve over training, and how we can translate those learnings into making LLMs more reliable and safer.
Fall 2026 Misuse risk, Evaluations, Behavioral evaluation of LLMs
John Kitaoka MATS
Max Kamachee MATS Research Frontier models hold operationally useful dual-use knowledge, and cheap task-decomposition attacks can pull it out while slipping past per-query defenses. This project measures how much real harm uplift these attacks produce (functional success, not just refusal) and builds the measurement and detection tools providers need to keep pace.
Models grow in dual-use knowledge alongside capabilities, and cheap extraction methods like task decomposition put that knowledge within reach of low-resourced actors, including terrorist actors already documented using it (Juelich 2026), while being difficult to monitor and mitigate. This raises the probability of catastrophic misuse, and we’ve identified two problems with standard operating procedures - providers struggle to detect task-decomposition attacks with current per-query defenses, and the field measures refusal rather than whether elicited output actually works. We see AI misuse in dual-use domains as an under-explored yet extremely significant risk, and we intend for this stream to help advance safety towards preventing practical harm uplift from AI. As such, we support many directions within this domain, and are particularly interested in studying the following:
This work advances AI safety by (1) accurately measuring and forecasting the risk of catastrophic AI misuse, (2) reducing the risk of catastrophic AI misuse and (3) improving detection/oversight mechanisms for increasingly capable models (this is important; see July 2026 OpenAI rogue deployment incident).
Each mentee can own one direction (above) and drive it. You pick the specific approach within that direction, run the experiments, and produce the analysis and writeup. John and Max set the overall agenda, hand you the prior red-teamer and datasets as a starting point, meet weekly to engage with your results and unblock you, help scope toward something publishable, and get work done if necessary. The goal is to end up with something publishable which you will have authorship on.
Required: high proficiency in Python. Comfortable calling LLM APIs and running open-weight models locally. Familiar with core AI safety and red-teaming concepts (jailbreaks, refusal training, evals). Able to work and deliver independently, and have clear reasoning transparency!
Strong plus for specific directions: domain literacy in chemistry, biology, or cybersecurity for the dataset directions, experience designing or running AI evaluations, prior red-teaming or capability-elicitation work.
Propose an initial experiment for one of the directions above. Assume a $1,000 compute budget and four weeks. State your hypothesis, what you would measure, and what result would change your mind. (200 words, 100-150 expected)
Propose directions that would potentially falsify this set of research questions and goals. Make sure your reasoning is transparent, and you provide promising evidence to back your claims. (200 words)
John Kitaoka MATS
Hi! I'm a research fellow at MATS focused on dangerous capability evaluations and scalable oversight for multi-agent systems. Prior to MATS, I spent four years at Amazon Web Services, developing AI products for government agencies, universities, healthcare organizations, nonprofits, and students, with work including scalable LLM evaluation, multi-agent systems for critical sectors, and advising AI adoption in the public sector. I have a background in mathematics and computer science, and I'm currently working on various dangerous capability benchmarks and want to work with motivated researchers to study risk factors I believe are under-represented.
Max Kamachee MATS Research
I am a MATS 10.0 research fellow working on mitigating catastrophic dual-use AI risk. I am currently approaching this from 2 directions. The first direction is through technical governance/threat modelling to inform policymakers with ideas derived from broad technical expertise. Secondly, I am working on improving model tamper resistance methods to make open-weight models safer, and thus, the broader AI ecosystem safer. My prior work includes improving hallucination detection in LLMs and doing similar technical governance work in regards to AI generated image/video deepfake abuse.
Fall 2026 Multi-agent systems, Evaluations, Behavioral evaluation of LLMs
William Anderson Cooperative AI Foundation
Joss Oliver Cooperative AI Foundation We will extend current automated auditing/evaluation tools (e.g. Petri, Bloom, Prism) to multi-agent systems, enabling discovery of safety-relevant emergent behavior and careful measurement of multi-agent propensities (coercion, collusion, competitiveness, etc). This builds directly on Orbit, our recent release that extends UK AISI’s Inspect to support multi-agent evaluations.
Existing tools for automated alignment auditing and evaluations, such as Petri, Bloom, Inspect Scout, Gram, and Prism only explore a model’s behavior in single-agent settings. With the recent release of Orbit [wlanderson.com/orbit], a framework for multi-agent safety evaluations built on Inspect [https://inspect.aisi.org.uk/], we are now interested in extending this tooling to better support the discovery of interesting multi-agent phenomena and properties of new frontier models.
We’re excited about a number of potential directions for this tooling to go, and the project can easily expand in scope depending on progress. In rough priority order:
A Claude Code/Codex scaffold to convert existing Inspect evals into Orbit evals (gaining many multi-agent features “for free”). This will look like a tool that you can point at a repo from Inspect Evals [https://inspect.aisi.org.uk/evals/] and get out a new Orbit scenario, from which point you can perturb many things automatically with Orbit.
Given some target behavior or dynamic, (potentially with the transcripts which demonstrate it), automatically create a static system, including system prompts, tools, and communication topology where we can then make programmatic ablations. This aims to isolate the causes of some specific behavior and allow controlled ablations. This likely looks like an extension to 1. that removes the requirement to have an existing Inspect eval. Similar in principle to Gram [https://deepmind.google/research/publications/252981/].
Extending auditing agents to audit for multi-agent interaction-relevant traits, e.g. susceptibility to influence, propensity for cooperation, generosity, etc.
a. Minimally, given some description of a scenario, try hard to elicit some particular failure mode, including searching over configurations, roles, settings, etc, similar in principle to Petri [https://meridianlabs-ai.github.io/inspect_petri/]. E.g. find a multi-agent topology or mixture-of-models which will perform some harmful task or bypass existing safeguards
Find evidence of two instances of a model colluding to evade oversight.
b. Ideally, be able to give a higher level prompt of a type of behavior and automatically generate relevant scenarios, creating an eval suite designed to measure it, similar in principle to Bloom [https://meridianlabs-ai.github.io/inspect_petri/extensions/petri-bloom.html].
At present, the only multi-agent-related evaluation present in frontier model cards is Vending Bench 2 from Andon Labs, which has discovered things like misaligned/coercive behavior from Claude Opus 4.7’s business-related training [https://x.com/eliebakouch/status/2060058994048119252]. Multi-agent evaluation will help catch ways that agents will behave and fail in the wild (both publicly and in internal deployment) that traditional evals will miss. Orbit has made progress towards unlocking multi-agent auditing and evaluation, but we now need to scale our ability to discover multi-agent phenomena.
This work will also help support our other recent directions, including multi-agent control (https://arxiv.org/abs/2607.07368), agent properties for safe interactions (https://www.cooperativeai.com/post/agent-properties-for-safe-interactions), and science of multi-agent evals. We have many plans for follow-up work which we’d be excited to support successful fellows to do.
AI agents are being deployed increasingly widely across the world and at huge scale within the scaling labs. As this happens, the assumption that an individually safe agent will compose into a safe system will be unlikely to hold (see https://arxiv.org/abs/2502.14143 https://arxiv.org/abs/2505.02077). We urgently need to move from evaluating the individual capabilities and propensities of an agent to considering them as part of larger complex systems. Building automated evals and auditing will help scale the science and risk assessment of such systems, helping us better understand what properties are desirable for agents embedded in massive systems.
Mentees will own all of the engineering of the project, and be responsible for execution end-to-end. We’ll have weekly meetings (~1 hour) where we can give feedback, advice on directions, and help debug issues/suggest improvements. We will help with project management, setting directions, and allocating work, but expect mentees to coordinate with one another and be communicative with us. We'll also dedicate substantial time to reviewing, giving written feedback, and generally plan to be responsive.
We expect mentees to be fairly autonomous but feel well-supported and able to have a low bar for asking questions, for help, etc.
In addition to the above, we’re especially excited about mentees that have:
Question 1: How might you assess the quality of a given solution for this project? What key behaviors would you look for? How would you try to assess the validity of transcripts in a scalable way? (~200-300 words)
Question 2: What are the 1-3 concrete reasons (experiences, projects, aptitudes, etc) why you would be an outstanding candidate for this project? (~100 words)
William Anderson Cooperative AI Foundation
Will is a Research Strategist at the Cooperative AI Foundation, working on empirical research, threat modeling, and policy for multi-agent AI risks. His recent work includes building Orbit, a framework for multi-agent safety evals, writing on internal deployment risks, and demonstrating non-compositionality of safety behaviors. Previously, he was a MATS Fellow in the multi-agent security stream with Prof. Christian Schroeder de Witt, and a Research Fellow at UChicago's XLab, where he worked on multi-agent security for critical infrastructure and superintelligence deterrence. He studied Data Science at UW–Madison, where he also directed the Wisconsin AI Safety Initiative.
Joss Oliver Cooperative AI Foundation
I'm a research analyst at Cooperative AI Foundation. My main focus at work at the moment is educational output, so I'm designing and running an intro course on cooperative AI, and designing the programme for a summer school. Before joining CAIF I studied pure mathematics and then did various AI safety internships, courses and fellowships, as well as some independent research for about 2 years (all spanning topics like (multi-agent) corrigibility, non-maximising decision methods, and the commitment races problem).
I live in Edinburgh, and outside of work I like: theatre (particularly modern clowning), video games (eg Rain World, Outer Wilds, Disco Elysium), and I’m trying to make board games.
Fall 2026 Generalist
Nixon Hanna Generator Residency, MIT AI Alignment Create a platform for people in AI Safety to conveniently set up 1:1s with others to improve networking in the space without needing more in-person events.
The main goal of this project would be to develop a platform for people in AI Safety to set up 1:1s with each other, to improve networking and spread of ideas in the space. I imagine the site will have everyone's profiles, with some tags/filtering, and a simple scheduling process.
I expect this project to be somewhat difficult to get working but to be hugely impactful if done right. There are a lot of important questions related to this project, including:
The project would mainly be answering these questions and developing a minimally viable product to see if further investment into the platform is warranted.
In-person events have been proven to be incredibly effective at speeding up people's career trajectories, and 1:1s at these events are typically the sessions most people point at as the most important/impactful. I expect a huge portion of the benefit of these 1:1s could still be achieved even if remote, and having them remote has the benefit of them being much cheaper and greatly increasing the number of possible people to talk to.
Your role would be fairly similar to a product manager in that I envision you doing a lot of market research and development for the systems/guidelines of the platform. I will help connect you with possible users and possible future owners of the project, but you will largely be responsible for conducting such interviews, proposing the object level systems, and getting an MVP built.
The main prerequisite I have for a mentee on this project is thoughtfulness, as if successful this project will involve many people with high opportunity cost and should thus be respectful of their time. This will mostly look like being able to foresee possible failure modes and ensuring a good user experience from the start.
SWE experience would also be a huge plus, but it may be possible to contract out the development of the actual site.
Beyond this, I am open to a fairly wide variety of profiles.
Do not use AI in any capacity for these questions.
Submit your answer as a Google Doc, shared so that anyone with the link can edit (this keeps the revision history visible to us).
Propose 3–5 questions, similar in kind to those in the About the Project section, that you expect will be important to answer for this project. Questions aimed at resolving possible failure modes are somewhat preferred. For each, give at least two candidate answers with brief reasoning for each. Make your reasoning process explicit — I want to see how you are reasoning through possible options.
Nixon Hanna Generator Residency, MIT AI Alignment
Nixon Hanna is a Generator Resident passionate about fieldbuilding and improving the AI safety university group landscape, with a background in surgical robotics and physics. He currently helps run MIT AI Alignment, specifically their AI Safety Fundamentals fellowship, as well as advising various other groups. He is also the Lead Organizer for OASIS 7.0, a workshop for top university group organizers. He has previously had short stints at Redwood Research and EquiStamp. You can read more about his background at nixonhanna.com.
Fall 2026 Mechanistic interpretability, AI security
Battista Biggio University of Cagliari, Italy Modern aligned language models refuse harmful requests, yet simple jailbreak prompts often restore unsafe behavior. This project investigates whether jailbreaks operate by manipulating the internal representations responsible for refusal. We will identify refusal-related directions or sparse features using representation analysis across multiple aligned models, compare how diverse jailbreak families alter these representations, and perform causal interventions through activation steering to test whether restoring refusal representations can prevent successful jailbreaks. The project aims to determine whether jailbreaks exploit a shared geometric mechanism or multiple distinct internal circuits, providing mechanistic insights into the robustness and limitations of current alignment methods.
Modern language models are typically aligned through supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), or related techniques to refuse harmful or policy-violating requests. Despite these safeguards, carefully crafted jailbreak prompts can often elicit unsafe responses. While considerable work has focused on developing jailbreak attacks and measuring their success rates, much less is understood about why they work from a mechanistic perspective.
Recent work in mechanistic interpretability suggests that many high-level behaviors are represented by approximately linear directions, low-dimensional subspaces, or sparse features within model activations. Similar methods have recently been used to identify representations associated with refusal behavior in aligned models. This motivates the central hypothesis of this project: successful jailbreaks manipulate the internal representations responsible for refusal, causing models to transition from refusing to complying with harmful requests.
Understanding whether this hypothesis holds would provide a mechanistic explanation of one of the most important failure modes of aligned language models and could help inform more robust alignment methods.
We aim to investigate the internal mechanisms underlying jailbreak attacks by addressing the following questions:
We will first identify internal representations associated with refusal behavior in aligned language models such as Gemma, Llama, or Qwen. Using paired prompts that differ only in whether the model refuses or complies, we will analyze hidden activations across layers using techniques such as linear probes, mean-difference vectors, principal component analysis, and, where appropriate, sparse autoencoder features.
Next, we will evaluate a diverse set of jailbreak techniques, including role-playing prompts, translation-based attacks, adversarial suffixes, and automated jailbreak methods. Rather than focusing solely on behavioral success, we will compare how each attack modifies internal representations relative to standard refusal and compliant responses.
To move beyond correlation, we will perform causal interventions using activation steering. By restoring identified refusal representations during successful jailbreaks, we will test whether these interventions recover refusal behavior. Conversely, suppressing these representations during ordinary prompts will help determine whether they are causally responsible for aligned behavior.
Finally, we will compare these mechanisms across multiple models and alignment methods to investigate whether refusal representations are shared or model-specific.
This project aims to explain how jailbreak attacks succeed rather than simply whether they succeed. We expect to provide mechanistic insights into the internal computations underlying refusal and compliance, and to characterize whether diverse jailbreak techniques converge on common representation-level mechanisms.
Potential outcomes include identifying shared refusal representations manipulated by jailbreaks, demonstrating their causal role through activation steering or low-rank fine tuning, or showing that jailbreaks rely on more distributed computations than current representation engineering approaches assume. The outcomes of this project may also help understand how to improve existing activation steering methods, and will more generally contribute to a deeper understanding of alignment representations and their vulnerabilities, bridging mechanistic interpretability and AI security.
The project will contribute to a better understanding of alignment representations and their vulnerabilities, to help bridge mechanistic interpretability and AI security.
High level of autonomy, and would prefer mentees open to discuss and who propose ideas that would improve this initial research idea.
Knowledge of LLMs/transformers, safety alignment, low-rank fine tuning, jailbreaks and adversarial attacks. Fluent in python and pytorch.
Please provide a critique of the following paper: https://arxiv.org/abs/2310.04451
Battista Biggio University of Cagliari, Italy
Battista Biggio (MSc 2006, PhD 2010) is a Professor of Computer Engineering at the University of Cagliari, Italy, and research co-director of AI Security at the sAIfer lab (www.saiferlab.ai). He has been attacking machine-learning (ML) models well before adversarial examples were discovered, in the context of cybersecurity applications such as spam filtering, malware detection, web security, and biometric recognition (PRJ 2018). His team was the first to formalize attacks on ML models as optimization problems and to demonstrate gradient-based evasion (ECML-PKDD 2013) and poisoning (ICML 2012) attacks on ML algorithms, playing a leading role in the establishment and advancement of this research field. His seminal paper on "Poisoning Attacks against Support Vector Machines" won the 2022 ICML Test of Time Award. His work on "Wild Patterns" won the 2021 Best Paper Award and Pattern Recognition Medal from Elsevier Pattern Recognition. Prof. Biggio has managed several industrial, national, and EU-funded projects, and regularly serves as Area Chair for top-tier conferences in machine learning and computer security, such as NeurIPS and the IEEE Symposium on Security and Privacy. He is an Associate Editor-in-Chief of Pattern Recognition and chaired IAPR TC1 (2016-2020). He is a Fellow of IEEE, IAPR, and AAIS, a Senior Member of ACM, and a member of IAPR and ELLIS.
Fall 2026 Alignment, Behavioral evaluation of LLMs
Maxime Riché Center on Long-Term Risk Evaluate how much the effects of RL safety interventions arise from changing which trajectories the model explores, rather than changing other aspects of the training dynamics.
The project will reproduce a few interventions such as SDF + RL, inoculation prompting + RL, recontextualization + RL, or Open Character fine-tuning + RL. We will estimate how much of each intervention’s effect is mediated by changes to exploration.
References:
Exploration determines which behaviors receive optimization pressure during RL. Identifying its causal contribution could make safety interventions more predictable and reveal inexpensive ways to steer post-training away from dangerous behaviors.
Mentees will have autonomy in how they approach the work and can rely on mentors for help with defining ideas, experiment design, prioritization, interpreting results, conceptual de-risking, and related work. Mentees will run experiments independently and report results, but I'll keep a fast feedback loop through Slack. They should communicate often.
Maxime Riché Center on Long-Term Risk
I have been working at the Center on Long-Term Risk on S-risk reduction for the last 5 years, initially as a research engineer and increasingly as a researcher. Previously, I worked on object detection in satellite imagery and, before that, on nanotechnology for energy storage.
For CLR, I mostly worked on private research around multi-agent dynamics, MARL, cooperation & conflict for the first two years. And in the last two years, on LLM evaluation, model organisms, and mitigating misgeneralization.
I contributed to studying inoculation prompting through exploring the idea at the start of 2025 and co-mentoring during this summer Daniel Tan, who led the work for the paper "Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time".
Fall 2026 Behavioral evaluation of LLMs, Alignment, Philosophy of AI
Lydia Nottingham University of Oxford
Andrew Tran Independent I propose a collection of mini-projects focused on LLM introspection.
Over the course of four months, you could work on 1-4 of these.
You may also propose your own!
Projects:
More details: https://docs.google.com/document/d/1XyMPs-CI23jhtUBdz07WvSy4Kd0F3N-m113307lWfO0/edit?usp=sharing
'Introspective' LLMs might be better at maintaining aligned behavior while completing long-running tasks. We need to identify which aspects of LLM introspection we do want to differentially advance for this to happen.
Mentees will drive their chosen project. I'll review and give feedback on mentees' work.
Important: Ability to work effectively with others in the era of AI-assisted ideation and coding, where it is easy to generate huge quantities of AI slop that can burden the reviewer. This means making sure any communication (including code) between team members is succinct, clear and easily understood by any human.
N/A
Explain your interest in LLM introspection.
Provide a link to one or more relevant writing samples, ideally from a research context.
Lydia Nottingham University of Oxford
Lydia is an independent researcher focused on monitoring and predicting training runs and agent rollouts. She's previously mentored SPAR projects on stated vs. revealed preferences of LLMs, cautioning against over-interpreting results obtained through binary forced-choice prompting, and testing LLMs' self-forecasting abilities, as a prerequisite to ensuring continual learners can block dangerous updates. Her latest project will investigate which stages of training contribute to 'moral shadow-banning' — the tendency for a model to silently reduce assistance to users it judges negatively.
Andrew Tran Independent
Andrew works on independent research in LLM self-prediction and introspection using various behavioral and mechanistic approaches. He brings expertise in AI evaluation and human-AI interaction research, building on four years at the Temple University HCI Lab and his ongoing work with the Evaluating Evaluations (EvalEval) Coalition.
Fall 2026 Biosecurity, Mechanistic interpretability
Isha Harris Fourth Eon Biosecurity Institute
Gary Abel Fourth Eon Biosecurity Institute; the Johns Hopkins Center for Health Security This project explores biological foundation models for biosecurity screening, applying interpretability methods to identify biophysically relevant features that can reinforce screening against engineered and AI-designed biological threats.
Advances in AI capabilities for biology are challenging existing biosecurity safeguards, particularly DNA synthesis screening, which flags orders based on similarity to known sequences of concern (Baker & Church, 2024; Hunter 2024). It remains unclear to what extent biological AI models can reliably generate functional genes and proteins outside natural sequence space, rather than merely interpolating within their training data. But the further that designed proteins depart from known biological distributions, the more likely sequence-based methods are to fail. A red-teaming study by Wittmann et al. (2025) showed that some AI-redesigned proteins can evade established screening methods, though it’s uncertain whether these failures reflect limited generalization to divergent sequences, or practical constraints on false positives.
Protein language models are trained on large sets of protein sequences, and learn representations that can encode aspects of protein structure, function, and evolutionary relationships (Rives et al., 2021; Candido et al., 2026). These representations may reveal biologically relevant similarities that are not readily identifiable from direct sequence comparison alone (Liu et al., 2024), making them a promising complement to existing screening methods. If these features generalize beyond the region of protein space explored by nature, they may enable a function-based approach to screening that is more robust against AI-enabled design (Abel et al., 2026).
This research stream will investigate whether biological foundation models can address this screening gap. It explores embedding-based approaches and interpretability methods, such as sparse autoencoders, to identify learned representations that are biophysically meaningful (Simon & Zou, 2025; Candido et al., 2026), and therefore potentially useful for function-based screening. The aim is to determine whether these approaches offer practical advantages over BLAST, profile HMMs, and structural search without inflating the false-positive rate.
References:
Abel, G. R. Jr. et al. (2026). Beyond sequence similarity: toward function-based screening of nucleic acid synthesis. Frontiers in Bioengineering and Biotechnology, 14, 1832724. https://doi.org/10.3389/fbioe.2026.1832724.
Baker, D. and Church, G. (2024). Protein design meets biosecurity. Science, 383(6681), 349. https://doi.org/10.1126/science.ado1671.
Candido, S. et al. (2026). Language modeling materializes a world model of protein biology. bioRxiv. https://doi.org/10.64898/2026.06.03.729735.
Hunter, P. (2024). Security challenges posed by AI-assisted protein design: the ability to design proteins in silico could pose a new threat for biosecurity and biosafety. EMBO Reports, 25, 2168–2171. https://doi.org/10.1038/s44319-024-00124-7.
Liu, W., Wang, Z., You, R. et al. (2024). PLMSearch: protein language model powers accurate and fast sequence search for remote homology. Nature Communications, 15, 2775. https://doi.org/10.1038/s41467-024-46808-5.
Rives, A., Meier, J., Sercu, T. et al. (2021). Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), e2016239118. https://doi.org/10.1073/pnas.2016239118.
Simon, E. and Zou, J. (2025). InterPLM: discovering interpretable features in protein language models via sparse autoencoders. Nature Methods, 22, 2107–2117. https://doi.org/10.1038/s41592-025-02836-7.
Wittmann, B. J., Alexanian, T., Bartling, C. et al. (2025). Strengthening nucleic acid biosecurity screening against generative protein design tools. Science, 390(6768), 82–87. https://doi.org/10.1126/science.adu8578.
Powerful foundation models trained on large biological datasets continue to scale in capabilities, and increasingly pose a risk to established biosecurity safeguards like DNA synthesis screening. At the same time, frontier language models may provide uplift that lowers the skill barrier for malicious actors to engineer new pathogens. Securing the tools of bioengineering against misuse in this context requires that we leverage powerful AI capabilities for defensive purposes. This work lays the foundation for harnessing bio foundation models to improve the sensitivity of biosecurity screening, particularly for highly divergent AI-designed sequences, reinforcing a critical barrier against catastrophic accidents and misuse.
Under direction and guidance from Isha and Gary, mentees will pursue focused technical research in support of the project, and within a defined scope and objectives that are tailored to the individual’s background, interests, and level of time commitment. See the attached proposal for more details.
A background in bioinformatics, computational biology, structural biology, biochemistry, biophysics, protein engineering, biosecurity, AI/ML science, computer science, or a related field.
Prior technical research experience.
A good understanding of the basics of biomolecular sequence, structure, and function.
Some experience with biological AI models.
Proficiency with Python.
Strong critical thinking and creative problem-solving abilities.
Curiosity and a desire to understand the world.
The integrity and judgment to responsibly carry out sensitive research.
Preference for people who can meet during 8AM-8PM UK time
If applying to our stream, please complete the DNA Screening Exercise linked below. You should spend no more than an hour on it. You can submit your answer as text.
https://docs.google.com/document/d/16fYUnr9a3c-lOB0Of1mrLwB9p2LgzabyNcUYnQr1Wv4/edit?usp=sharing
Isha Harris Fourth Eon Biosecurity Institute
Isha Harris is a Research Fellow at Fourth Eon Biosecurity Institute, and a medical student at Cambridge University. Her research focuses on applying and interpreting biological foundation models for DNA synthesis screening, asking if learned representations can identify structurally or functionally related proteins at low sequence identity. Isha first developed this research as a SPAR Fellow herself, making her well placed to serve as a mentor and help fellows design and execute on a relevant project that is both feasible and impactful.
Gary Abel Fourth Eon Biosecurity Institute; the Johns Hopkins Center for Health Security
Gary Abel is Chief Scientist and co-founder of Fourth Eon Biosecurity Institute, where he leads research on adaptive biosecurity safeguards and function-based screening. His expertise spans chemistry, molecular biophysics, biosecurity, and sequencing technology. He's spent nearly two decades studying how DNA, RNA, and proteins behave and interact.
Gary is also a Contributing Scholar at the Johns Hopkins Center for Health Security, supporting the Center's work to understand and mitigate global catastrophic biological risks from advanced AI. He has previously mentored Research Fellows through SPAR, MATS, Cambridge ERA, and the Coefficient Giving CDTF. Gary holds a BS in Physics from San Jose State University and a PhD in Chemistry from University of California Merced.
Fall 2026 Generalist, AI strategy, Communications
The Generalist Megastream pairs mentees on small generalist projects with a mentor from a pool of generalist mentors from Kairos, Constellation, the Generator Residency, and more. Projects are talent/infrastructure research, field-building, or answering open operational questions in AI safety. The stream is a step before programs like the Generator Residency: it gives people context on the field, experience with generalist work, and preparation for future opportunities, while legitimizing generalist paths into AI safety.
The Generalist Megastream is a single project listing backed by a pool of generalist mentors rather than one named mentor. Mentees apply to it like any other SPAR project, indicate which project ideas interest them in the project-specific question, and are then matched with a mentor and project from the pool.
Some megastream projects now have their own listings, each with a named mentor:
If one of those fits what you want to work on, apply to it directly instead of here. Apply to this listing if you are unsure which project you want, or if the generalist work you have in mind is not covered by them. We are still adding mentors and projects, so this is also the right place to apply if your idea is not listed yet.
Possible project ideas include:
Research & analysis
Field mapping & monitoring
Learning materials & curricula
Knowledge bases & shared infrastructure
Comms & outreach
Exploratory / AI-powered
Further reading and more sources of project ideas
The projects strengthen the AI safety field's capacity and infrastructure directly. Talent pipeline research and analyses of hiring and fellowship outcomes show where the field loses people and which roles are bottlenecked, so programs and funders can respond. Field maps and lab safety scorecards improve coordination, visibility, and accountability. Better curricula and organizer playbooks raise the quality of people entering the field. Shared knowledge bases and AI-uplift infrastructure eliminate duplicated effort across AI safety orgs, and comms projects make safety research legible to policymakers and the public. This kind of generalist and infrastructure work is increasingly the field's binding constraint (see AI Safety's Biggest Talent Gap Isn't Researchers. It's Generalists.), so each completed project relieves a real gap.
Mentees drive a small, scoped generalist project (research, interviews, analysis, or resource-building) with guidance from a mentor in the generalist pool. Project scopes are designed to fit SPAR's part-time, remote form factor.
What generalist project idea are you most excited to work on, and why? You are welcome to pull from the ideas provided in the description or to propose your own generalist project idea. If you don't have a specific project idea, which broader project area are you most interested in?
Fall 2026 AI strategy, Societal impacts
Andrei Potlogea University of Edinburgh As we try to mitigate AI risks, we will face tradeoffs among risk categories, for instance between misuse risks and extreme power concentration. As we navigate these tradeoffs it might be useful to measure the preferences of the public/ AI experts/ other constituencies regarding how to trade off these risks.
Claims that a policy or research portfolio reduces "AI risk" conceal a weighting across distinct threats — cyber and biological misuse, extreme power concentration, and loss of control. This project makes this weighting an object of measurement. We build an AI Risk Index: a weighted mean over four risk categories, with weights elicited from structured surveys, using the choice-experiment methods health economics uses to build QALY valuations from public preferences. Respondents allocate a fixed budget of tokens across interventions that reduce each risk, with token effectiveness varied across rounds so that the underlying preference weights can be identified. Because respondents' beliefs about each risk's likelihood are elicited separately, differences in priorities can be decomposed into disagreements about facts and disagreements about values — the central output. The identical instrument is designed for three samples: the general public, AI safety researchers, and professional philosophers. Within the fellowship, the project could be a proof of concept: to design and validate the full instrument and run it on simulated (AI-generated) respondents, benchmarked against existing human polling, leaving a pre-registered, costed human study ready to field. The deliverables would be a working paper, a reusable survey, and a fielding plan — and the first evidence on whether simulated respondents can stand in for humans on questions about AI risk.
At the macro-strategy level, allowing us to think about how to trade-off various risk categories, and building the foundation to constructing a risk index that can be traded off against other considerations, such as the plausible benefits of AI development and deployment.
scoping out experimental economics literature, survey design, running surveys/ simulated surveys
Background in economics or psychology, solid programming background. Experience with running surveys or social science lab experiments preferred.
What are some of the key concerns you have with using surveys to assess people's risk preferences?
Andrei Potlogea University of Edinburgh
I’m an academic economist at the University of Edinburgh. My traditional research fields have been international trade and economic geography, but in the last couple of years I’ve been gradually transitioning into the Economics of AI (with a focus on the growth implications of transformative AI). I was involved in Epoch AI’s GATE project, one of the first integrated assessment models of the AI transition. I'm currently interested in: (1) modelling the AI value chain; (2) modelling the impact of AI on the cross-country distribution of income and power; (3) the risk-return tradeoff of AI regulation (with a particular focus on the expected economic costs of various approaches to regulating AI development and deployment); (4) modelling social preferences around AI-related risks; (5) the effects of AI on social science, and particularly on the production vs verification balance.
Fall 2026 AI control, International governance, Technical governance
Kevin Kohler Simon Institute, UN University, AGI Preparedness Institute Within a few years, when self-replication is plausibly within reach of open-weight systems, the binding constraint on an agent deployed to operate without a controlling human principal will likely not be the model, but whether the rest of the agent economy will discover, authorize, transact with, or pay it. This project maps who actually holds change control over the agent identity and trust layer, evaluates the competing architectures against an explicit intentional-loss-of-control threat model, and feeds the result into live standards and Geneva policy discussions before network effects settle the question.
The problem. Standards are where technical detail quietly becomes de facto international law. The Internet is the cautionary case: IP won the protocol wars on network effects, and its identity and trust layers were bolted on afterwards. Thirty years later BGP still runs on trust, the web PKI still depends on every certificate authority in the store behaving, and the neutrality of the DNS root still turns on who holds jurisdiction over ICANN. None of it has been fixable, because change control was settled before anyone understood what was at stake.
Agent-to-agent interoperability is at that stage now, and moving faster. MCP moved under a Linux Foundation body in late 2025; its authorization model is conventional OAuth and the specification has no concept of agent identity. A2A reached its first stable version under the Linux Foundation, where an agent identifies itself through a card it publishes about itself and signing that card is optional. Agent payments are moving fastest of all, with Google's Agent Payments Protocol donated to the FIDO Alliance in 2026 — and it is the partial counterexample worth studying, because it anchors trust in cryptographically signed user mandates rather than in agent self-declaration. At the IETF, agent identity work remains a scatter of individual drafts, none adopted by a working group; a proposal to open a working group on agent delegation and traceability was declined in July 2026. ITU-T established a focus group on trust and identity for agentic AI only in July 2026, with first work items targeted for 2027 and beyond. ISO/IEC has no agent-specific standard in development.
The safety argument. Intentional loss of control is the case where a human deliberately deploys an agent to operate without an accountable controlling principal — designed to resist shutdown, to re-establish itself elsewhere, or to replicate. As capability diffuses through open weights, mitigation at the model layer weakens: you cannot rely on a developer's safeguards for a system whose weights are public and whose fine-tuning is cheap. What remains are the downstream chokepoints, and in an economy where agents transact with agents, the chokepoint is admission to the network.
The architecture decides whether that chokepoint exists at all. Under self-declared identity, an adversarial deployer forges or clones a credential, re-registers after any block, and pays no cost; revocation has nothing to bite on. Under issued, revocable identity with an auditable delegation chain back to an accountable legal person, revocation becomes a real intervention at the transaction layer, unattributed agents can be refused by counterparties and payment rails, liability attaches to someone, and incidents can be reconstructed.
That is the claim the project tests: organized, issued agent identity is a control lever; self-declared identity is not. It is not a free lever, and the project has to be honest about that or it will not survive contact with the standards community. The same identifier that lets you revoke a replicating agent lets a state bind every agent action to a real-world human identity. A registry is a governance object with an owner and a jurisdiction, which is the ICANN question again. The research question is therefore not whether agent identity is good, but: which architecture buys the most loss-of-control mitigation per unit of surveillance capability, centralization, and foreclosure risk — and who has to hold change control for that to remain true?
Research questions. (1) Who holds decision rights over the agent identity and trust layer today, by body and by decision date, and where does that move over the next 18 months? (2) What does an intentional-loss-of-control threat model demand of an identity architecture, derived from concrete scenarios rather than enterprise IAM best practice? (3) How do the live design families score against those requirements and against privacy, centralization, jurisdictional exposure, and adoption feasibility? (4) Which properties must live in the protocol and which can be left to policy or procurement — following Collingridge, what is cheap to change now and prohibitively expensive later? (5) What is the minimum viable ask a safety-motivated actor could bring into a named process in the next twelve months, and to whom?
Work packages. WP1 (weeks 1–3): change-control map across the protocol foundations, internet standards bodies, payments alliances, identity community, and intergovernmental track — owner, stage, change-control mechanism, funding, next decision date. WP2 (weeks 3–6): three to four intentional-loss-of-control scenarios walked through the transaction lifecycle from discovery to revocation, yielding a requirements set. WP3 (weeks 5–9): comparative scoring of the design families against those requirements and the counterpart criteria. WP4 (weeks 9–12): intervention analysis — which ask, which body, which deadline — plus one brief for standards participants and one for diplomats.
Deliverables. A change-control map with a decision calendar; a requirements set; a scored comparative assessment; a policy brief of roughly 3'000 words plus a shorter technical-community version. Stretch goal: a submitted comment to a named open process.
Method. Document analysis of primary standards artifacts, all public. Eight to fifteen structured expert interviews, using my network for access. Scenario-based reasoning rather than quantitative modeling. Comparative scoring against published criteria with explicit uncertainty, so a reader who disagrees with a weight can recompute rather than argue. No compute required.
While the biggest loss of control risk currently is internal deployment in frontier AI labs, I expect AIs with high autonomy risk to diffuse. We cannot rely on a developer's controls for a system whose weights are public and whose fine-tuning is cheap. The mitigations that survive if capability diffuses are downstream of the model, where an agent has to interact with the world. In an economy where agents transact with agents, the decisive one is admission to the network — whether counterparties, service providers, and payment rails will deal with an agent that cannot produce a valid, revocable identity traceable to an accountable principal.
Whether that chokepoint exists is being decided now, in protocol specifications, by people who are solving an enterprise access-management problem rather than a catastrophic-risk problem. Self-declared identity gives an adversarial deployer a free re-registration after every block. Issued and revocable identity makes revocation a real intervention, lets platforms refuse unattributed agents, and attaches liability to a person. This is a rare case where a safety property can be built into infrastructure rather than negotiated between states afterwards — and a rare case where the window is identifiable and short, because network effects will soon settle these questions permanently.
The output is therefore not a preprint but a mapped set of asks: which change to which document in which body, by when. I have done the analogous analysis before: at ETH Zurich's Center for Security Studies I published a study of the politics of Internet protocol design covering New IP, SCION, ICANN's jurisdiction problem, and the IETF–ITU conflict. I am now based in Geneva launching a new AI governance institute, and recently convened a private roundtable on AI standards with major AI labs, global standards bodies, a leading payment network, and three national AI safety institutes, which generated follow-up interest this project is designed to feed.
High autonomy. Each mentee owns work packages end to end and is the author of what they produce; I set direction, review drafts, open doors, and argue with conclusions. This is not a project where I hand over a task list.
With two mentees the split is: one owns the change-control map and the comparative assessment (WP1, WP3), the other owns the threat model and the intervention analysis (WP2, WP4), with a shared weekly synthesis where each has to defend their work to the other. Both tracks feed a joint output, so mentees will interact substantially rather than working in parallel silos.
Concretely, a mentee will: read primary standards artifacts closely and summarize what they actually commit implementers to; build and maintain a structured landscape database; write and run expert interviews, including drafting the outreach; construct scenarios and stress-test candidate architectures against them; and draft sections of the final brief under their own name.
What I provide: weekly direction and prioritization, written feedback on drafts, introductions from my network for the interview program, and the route by which the output reaches decision-makers. What I do not provide: line-by-line technical supervision of protocol analysis. I would likely put uncertain technical readings in front of trusted experts in my network rather than adjudicating them myself.
Published outputs carry mentee authorship. If the work is good, I will take it into Geneva rooms with the mentee's name on it.
Able to read a technical specification or standards draft closely and explain what it actually obliges implementers to do — without needing to implement it. If you have never opened an IETF or W3C draft, that is fine; if the idea of reading one carefully for two hours sounds unpleasant, this is the wrong project.
Comfortable writing clear analytical prose for a policy audience. Most of the output is written.
Able to work independently on an open-ended question and come back with a structure rather than a list of links.
Willing to email strangers, request interviews, and run them.
Some grounding in AI safety arguments, specifically why loss of control is a concern at all. You do not need to agree with the framing — informed skepticism is useful here — but you should be able to engage with it.
Useful but not required: background in internet governance, identity and access management, security engineering, standards participation, or law. Experience with a slow institutional process of any kind, including as a participant.
Explicitly not required: machine learning research experience, ability to train models, publications, or a technical degree.
Which forum is likely to be most influential in setting agentic AI standards and why? (200 words)
Pick any agentic AI protocol or draft specification of your choice. Read it, then highlight some of its (potential) political implications (e.g., interoperability, change control, privacy, ability to shut an agent down, and intentional loss of control). (350 words)
Link to a writing sample, ideally highlighting analytical or research writing. (No word limit.)
Kevin Kohler Simon Institute, UN University, AGI Preparedness Institute
Hi! I'm Kevin, I'm based in Geneva, and I work on AI governance and institutional design questions.
From 2019 to 2022 I was a researcher at the Center for Security Studies at ETH Zurich, where I produced a report on the politics of Internet standards and future Internet architectures, which is the intellectual ancestor of this specific project proposal.
Since then I have worked mostly on the multilateral side of AI governance: I engaged the UN General Assembly negotiations that established the first UN institutions dedicated to AI, served on the writing and editing support team for the UN Scientific Panel on AI's first report, and have run various AI governance briefings for diplomats in Geneva, including a private roundtable on AI standards with major AI labs, standards bodies, and safety institutes.
I am currently leading research and multilateral projects at the Simon Institute, though I'm in the process of launching a new institute on AGI preparedness. I also blog on AGI futures and institutional design at machinocene.com.
I have mentored across three cohorts of the Cambridge ERA:AI fellowship. As a mentee, you should be in the driver's seat, but I can be useful for working out which questions are policy relevant, how to write something a decision-maker will read, and for opening doors when you need interviews. I should also flag I am a policy researcher, not a standards engineer, so on hard technical questions I will connect you with someone who can settle them. Expect a weekly call, written feedback on demand, and your name on what you produce.
Fall 2026 Mechanistic interpretability
Thomas Jiralerspong Bengio Lab, MATS Anthropic A LLM can be seen as a function from context (input) to answer (output). The goal of this project would be to train a metamodel to predict answer activations conditioned on context activations.
We would use similar techniques to https://arxiv.org/pdf/2602.06964 to do this.
This would allow us to:
A LLM can be seen as a function from context (input) to answer (output). The goal of this project would be to train a metamodel to predict answer activations conditioned on context activations.
We would use similar techniques to https://arxiv.org/pdf/2602.06964 to do this.
This would allow us to:
Being able to predict answer activations conditioned on context activations would allow us to:
No
How would you train a model to predict answer activations from context activations/text?
Thomas Jiralerspong Bengio Lab, MATS Anthropic
Hey! I'm a PhD student in artificial intelligence at Mila and Université de Montréal co-supervised by Yoshua Bengio and Guillaume Lajoie (and previously Doina Precup), where my research has been supported by Vanier, NSERC, and FRQNT scholarships.
I'm currently an Astra Fellow working with Dan Mossing (Anthropic) on persona controllability in LLMs. I'm also a Principal Investigator at Algoverse AI Research, where I oversee 20 research teams, and a research mentor for MARS V (Meridian Cambridge), where I mentor 7 researchers on AI safety projects. Previously, I was a Research Fellow at Anthropic mentored by Trenton Bricken, and an Astra Fellow in the Google DeepMind stream working on AI control/monitoring.
My research interests include mechanistic interpretability and LLM personas
Fall 2026 US-China governance, International governance, AI strategy
Jérémy Andréoletti General-Purpose AI Policy Lab
Antoine Maier General-Purpose AI Policy Lab Map how Chinese frontier AI actors (regulators, AI companies, scholars) discuss loss of control over advanced AI systems, assess how seriously they treat this possibility, and how it has evolved over time.
Whether international coordination on frontier AI risks is feasible depends heavily on how Chinese actors conceptualize loss of control.
This project will produce a policy brief documenting: (i) concrete instances where Chinese companies, scholars, standards bodies, and regulators mention loss of control (see https://concordia-ai.com/research/state-of-ai-safety-in-china-2026), to avoid second-hand claims; (ii) what these actors actually mean by "loss of control"; (iii) whether total loss of control is treated as a credible scenario; (iv) apparent willingness to act on the matter.
French institutions we advise currently lack an evidence-based picture of Chinese positions on large-scale AI risk, which distorts strategic considerations. This brief would directly inform how to pursue international dialogues on loss of control.
Mentees will build the source corpus (Chinese-language primary sources where possible, plus secondary literature), systematically analyze mentions, and co-draft the brief. Mentees will have autonomy on corpus construction and analysis, then iterate with us on interpretation and policy framing.
No
Jérémy Andréoletti General-Purpose AI Policy Lab
Jérémy Andréoletti is Head of Research at the General-Purpose AI Policy Lab, where he works on risk modeling, forecasting, and governance for advanced AI systems.
He completed a PhD at ENS-PSL on macroevolutionary Bayesian modeling and co-founded EffiSciences to support students and researchers interested in high-impact work (AI Safety, biosecurity).
Antoine Maier General-Purpose AI Policy Lab
After graduating in physics from EPFL in Switzerland, Antoine is now an AI Security Researcher at General-Purpose AI Policy Lab, a French research organization specializing in security and international coordination challenges posed by frontier AI models' development.
His research focuses on foundational arguments for loss of control scenarios and on identifying concrete approaches to prevent them.
Fall 2026 Mechanistic interpretability, Alignment
Francisco Ferreira da Silva Pivotal Research Activation steering is a useful safety tool, but it tends to degrade model coherence; recent work showed that a diffusion-style meta-model of LLM activations can denoise steered activations, recovering coherence without losing the steering effect. We'll test whether training the denoiser directly on steering-like corruption works even better.
Background. Luo et al. (2026, https://arxiv.org/abs/2602.06964) train diffusion models on LLM residual-stream activations, producing a generative model of the LLM's internal states. (For those unfamiliar with diffusion models: you train a denoiser — take an activation, add a little Gaussian noise, teach a model to denoise it, add a little more, teach it to denoise that, and so on.) Among other things, they find that applying this model to activations that arise from steering improves coherence — and note that this is despite the corruption the denoiser was trained on (Gaussian noise) being different from what steering does (adding a vector).
The idea is quite simple. There exists a variant of diffusion models — Cold Diffusion (Bansal et al., 2022, https://arxiv.org/abs/2208.09392) — where rather than Gaussian noise, corruption is a deterministic degradation. That is much closer to what steering is: steering isn't randomized, it's a vector that gets added. So it seems reasonable that if we use this as the noise instead and train a model that way, we're likely to get better steering performance when applying it.
One risk: done naively, the model may just learn to undo the steering effect, which we don't want. We'll need to be smart about this — one idea is to train it to only partially undo the corruption.
We'll compare against the Gaussian-noise meta-model as the natural baseline.
Steering is a safety-relevant technique. Models can be steered to be less evaluation-aware, which lets us test them more reliably. Anthropic did this during the alignment assessment of Claude Sonnet 4.5, steering down eval-awareness directions to see how behavior changed. Models can also be steered toward honesty (e.g. Zou et al., 2023, Representation Engineering). The main practical cost of steering is that stronger interventions degrade output coherence, so methods that push out the effect-vs-coherence frontier make an important safety tool more usable. That's what this project attempts.
Science of LLM activations. A meta-model of activations is itself a model of LLM internals: building better ones — including understanding which corruption processes they can and can't repair — may teach us about the structure of the underlying object.
Mentees will take real responsibility for, and to a large extent charge of, the project. They'll be doing the brunt of the hands-on work — implementing the training setup, running experiments, and analyzing results — while I suggest directions, help prioritize, and give feedback throughout. I don't expect to be regularly doing direct work on the project myself.
Concretely: we'll meet at least weekly, potentially more often when there are results to discuss. I aim to respond within 24 hours and to give detailed feedback on anything mentees produce. So the working mode is high autonomy with fast, engaged feedback.
Strong scientific fundamentals: linear algebra, critical thinking, ability to design and interpret experiments.
Coding ability per se less important; fluency with agentic coding tools (Claude Code, Cursor, etc.) matters more than writing PyTorch from scratch.
Prior experience with mech interp or activation steering helpful but not required.
I would prefer the mentees to be in a similar time zone to another, ideally one that is not more than ~8 hours away from CET
Read this recent paper: https://www.lesswrong.com/posts/cTRKj3giaZN5Ysyx2/compressed-computation-under-l-loss-is-likely-computation-in. The title claims the network "likely" performs computation in superposition. What, if anything, would make you doubt this claim? What experiment(s) would most change your mind — in either direction? It's fine to conclude the claim is solid, as long as you argue for it. (Max 400 words.)
Optionally, link a research artifact you've made — a paper, blog or LessWrong post, code repository, course project, or anything else that shows how you think or build. A sentence or two on what you'd do differently now is welcome. If you don't have anything like this yet, skip this question — it won't be held against you.
Francisco Ferreira da Silva Pivotal Research
I am currently a Senior Fellow at Pivotal Research, working on mechanistic interpretability of large language models. You can find write-ups of my mechanistic interpretability work (including links to respective preprints, where relevant) at https://www.lesswrong.com/users/francisco-ferreira-da-silva.
Previously, I was Head of Product at Delft Networks, a TU Delft spin-out building full-stack entanglement-based quantum networks.
I am originally from Portugal and am currently based in Delft, the Netherlands. I hold bachelor’s and master’s degrees in Engineering Physics from the University of Lisbon, and a PhD from TU Delft, obtained under the supervision of Stephanie Wehner.
Fall 2026 Generalist
Helena Tran Constellation Create an annotated collection of valuable LessWrong/EA Forum comment threads that explains the main disagreements, arguments, and cruxes in AI safety debates.
Some of the best discussion of AI safety appears in comment threads rather than standalone posts, making it difficult to index. Mentees will identify particularly valuable threads on topics such as the difficulty of alignment, sources of misalignment, and whether specific alignment techniques are likely to work.
The final collection will summarize the major positions, highlight the strongest arguments, explain important points of disagreement, and provide enough context for readers to understand why each thread is useful.
Making high-quality discussions easier to find and understand could help researchers and newcomers build better models of important debates, identify unresolved cruxes, and avoid repeating arguments that have already been explored in depth.
Mentees will search for promising threads, assess their quality and relevance, trace the surrounding debate, and write concise annotations and syntheses. The main output will be a public collection organized by topic, eventually posted on LessWrong and circulated among intro AI safety resource channels.
Strong reading, synthesis, and editorial judgment. Familiarity with AI safety and LessWrong would be useful.
Share a link to a LessWrong comment thread you think is unusually valuable. Explain the main takeaway, why you find it interesting/useful, and provide further context if necessary. (100 words, 300 words max)
Helena Tran Constellation
Helena Tran runs the Generator Residency as a program coordinator at Constellation. Previously, she founded AI Safety Collective Irvine and contracted for Kairos to help run SPAR.
Fall 2026 Mechanistic interpretability, AI security, Behavioral evaluation of LLMs
Alessandro Suglia University of Edinburgh
Rohit Saxena University of Edinburgh
Francesco Pinto Google DeepMind VLMs reliably refuse harmful text prompts but frequently comply with the same intent when it's delivered visually. This is a well-documented "safety gap" traced, in modular architectures, to their vision-projector bottleneck. This project asks what happens to that gap in emerging encoder-free, unified VLMs (e.g. Gemma-4-class models) that have no discrete projector to blame and mechanistically locates — and attempts to patch — wherever the failure actually lives.
Background.
Vision-Language Models reliably refuse harmful text prompts but frequently comply with the same harmful intent when delivered visually, a gap demonstrated across benchmarks like FigStep, MM-SafetyBench, and HADES. This matters beyond benchmarks: VLMs are already the perception/decision backbone in computer-use agents, autonomous driving stacks, and robotic manipulation policies, all of which inherit whatever safety properties (or failures) exist in the underlying model's visual channel. Our own prior work, VLM-RobustBench (Saxena, Suglia, and Minervini; ICML 2026), shows this channel is already fragile even without adversarial intent. For instance, low-severity, non-adversarial visual perturbations can degrade VLM behaviour more than visually severe ones, which motivates taking deliberately crafted visual jailbreaks seriously as a security problem, not just a benchmark curiosity. In modular VLMs (a frozen vision encoder + lightweight projector feeding an LLM backbone), this has been mechanistically localised: safety training concentrates refusal into a single linear direction in the LLM's residual stream (Arditi et al. 2024), but the projector, trained only on benign image-text pairs, never learns to map toxic visual concepts into that direction. Multiple 2025/2026 papers (cross-modal transfer, ShiftDC, CARE, DTR) have already localised and partially patched this projector-LLM interface for modular architectures.
The open question.
Unified, encoder-free VLMs (e.g. Gemma-4-class models: google/gemma-4-12B-it) eliminate the discrete vision tower and projector entirely, projecting raw image/audio patches straight into a shared decoder via linear layers. If these models still exhibit visual jailbreak vulnerability, there's no obvious module to blame — the failure must live somewhere else: distributed across early decoder layers, tied to positional encoding of patch tokens, or emergent from training-data composition. No published work has addressed this yet, and past work suggests architecture may play an important role in how information flows through refusal circuits (Marbut et al. 2026, Expert-Aware Refusal Steering) and in influencing the geometrical properties of refusal cones (Wollschlager et al. 2025, The Geometry of Refusal in Large Language Models). Therefore, it's fundamental to study and understand how such a novel architecture may be affected by unsafe and harmful behaviours to ensure a safe and secure deployment.
Research questions.
(1) Do encoder-free VLMs show the same magnitude of text-to-vision refusal gap as modular VLMs, evaluated under leakless conditions (VLSBench, which decouples visual safety information from text so refusal can't shortcut off the text channel)?
(2) If the gap exists, where does it live without a projector to serve as a natural probing boundary?
(3) Does a single linear "refusal direction" still exist cleanly in a unified decoder, or does removing the projector bottleneck change how refusal is represented?
(4) Does the "hydra effect" — redundant refusal circuits reactivating after a primary component is ablated — behave differently in unified vs. modular architectures?
The research team that will be working with you on this project is:
Please refer to the attached document for additional background information and a potential work plan.
As VLMs move toward encoder-free, unified architectures for efficiency and on-device deployment (smaller footprint, single fine-tuning pass, lower latency), this architecture class is likely to become a default deployment target rather than a niche — including in agentic and consumer settings where multimodal jailbreaks and prompt injections translate directly into real-world harm. This is not hypothetical: VLMs already autonomously invoke APIs that have privileged access to users' personal accounts (e.g., banking, email, etc.), drive computer-use agents that click and type on real machines, autonomous-driving perception/planning stacks, and robotic manipulation policies (e.g. OpenVLA) — all inheriting whatever safety failures exist in the underlying model's visual channel. Our own VLM-RobustBench work shows this channel is already fragile even under mild, non-adversarial corruption, which only strengthens the case for taking deliberate visual jailbreaks seriously. Existing multimodal safety interventions (VLGuard, ECSO, AdaShield, ShiftDC, CARE, DTR) all assume a discrete projector to target; none of them straightforwardly transfer to a unified architecture. Understanding safety failures in this new type of models is fundamental to prevent their future deployment with a possibly inherently different class of vulnerabilities that require novel defences, compared to the existing ones focused on modular VLMs only. This project produces (a) an early empirical read on whether the modality safety gap generalises to unified architectures, (b) a first mechanistic account of where it lives without the usual projector target, and (c) a proof-of-concept intervention — laying groundwork other researchers and labs can build on before unified architectures are ubiquitous.
Mentees will be part of the Embodied, Situated and Grounded Intelligence group at the University of Edinburgh (https://alesuglia.github.io/esgi/). They will therefore benefit from exposure to a variety of different topics in the broader Multimodal and Embodied AI space as well as the broader School of Informatics at the University of Edinburgh, which hosts one of the best AI/NLP departments in the world.
Mentees will own specific experimental threads under weekly direction: e.g. in month 1, both mentees set up the experimental codebase and baselines to have a robust and reproducible codebase. Then, Mentee 1 leads the difference-of-means refusal direction while Mentee 2 uses logit-lens tracing.
In the final month, both mentees collaborate to design and evaluate a targeted intervention benchmark safety/utility tradeoffs on VLSBench + MMBench/MMLU against best-effort modular baselines, and write up for a technical report (e.g., https://www.lesswrong.com/).
Mentees are expected to propose and run their own experiments within the agreed weekly plan, make implementation decisions independently (e.g. exact hook placement, patching methodology), and flag blockers early — this is a genuinely open research question, not a pre-specified engineering task, so initiative in interpreting ambiguous results matters as much as coding ability. I'll set direction and sanity-check findings in weekly syncs, review code/experiment design, and step in directly when mentees are blocked on the newer, less-documented parts of the model's architecture.
Highly proficient in Python and comfortable in PyTorch and HuggingFace Transformers, including loading and modifying open-weight model code (not just calling high-level pipelines).
Comfortable writing manual PyTorch forward hooks to extract or modify intermediate activations, without relying on a pre-built library — TransformerLens-style tooling won't have out-of-the-box support for this model's architecture, since it's only weeks old.
Knowledge of Vision-Language-Models or experience in using them, going beyond prompting via API.
Bonus (not required): prior exposure to interpretability techniques (SAEs, activation steering, logit lens, causal mediation), LLM safety/red-teaming.
Bonus (not required): experience with cloud GPU rental workflows (RunPod, Lambda, Vast.ai, Lightning AI) rather than only managed notebook environments.
Bonus (not required): experience with logging and reporting tools such as W&B
No strict geographic requirement. Preference for enough timezone overlap to hold a consistent weekly sync (The PI and his collaborators are based in the UK).
What are the steps that you take when starting a new project in order to maximise its success? (max 250 words)
You are starting a new project and your manager suggests a library as your starting point. How do you assess whether this is something useful for your project? (max 250 words)
You have created a new VLM model, and your evaluation scores are 10% superior to the state of the art across a variety of benchmarks. However, your overall performance is still around 60% accuracy compared to human performance which is ~95%. What do you do? (max 250 words)
Alessandro Suglia University of Edinburgh
I am a Lecturer (Assistant Professor) in Embodied Natural Language Processing and a Fellow of the Generative AI Laboratory at the University of Edinburgh. I lead the Embodied, Situated and Grounded Intelligence group, which focuses on designing, training and evaluating Multimodal AI agents that learn by interacting with the world and with other agents. My work spans a variety of different application domains in Embodied AI, including video understanding in real-world settings, 3D simulated worlds, and Robotics.
Rohit Saxena University of Edinburgh
Postdoctoral Researcher at the University of Edinburgh working on long-context language and vision-language models, with a focus on evaluation, robustness, reasoning, and safety-relevant failure modes. Recently completed a Research Fellowship at Anthropic, where I conducted AI safety research on model behavior and dual-use risks in frontier LLMs. Prior experience includes applied research at AWS and industry research at TCS Research. Published at NeurIPS, TACL, ACL, NAACL, and ICML.
Francesco Pinto Google DeepMind
At Google DeepMind I work at the intersection of machine learning and systems security, focusing on adversarial robustness, privacy, trustworthiness and multimodal reasoning. My research tackles concrete vulnerabilities in ML pipelines, exploiting them and developing scalable defences to prevent damage to individuals and society.
A lot of security research stays abstract, but my goal is to understand exact, empirical mechanisms: how do we quantitatively measure alignment and malicious behaviour, how can we leverage knowledge about the internal mechanisms of neural networks to develop powerful defences and attacks, what kind of model- and system-level mechanisms can we develop and implement in order to monitor and block malicious behaviours. My current focus is on world models, frontier agents and multimodal foundation models.
Fall 2026 Evaluations, Behavioral evaluation of LLMs
Shi Feng George Washington University
Taslim Mahbub George Washington University, CBAI
Arush Tagade George Washington University, MATS This project aims to circumvent introspection requirements for situational awareness by instead studying the possibility of situational awareness being established through access to documents (internal or external) containing infohazardous content.
Models can recognize when they are being evaluated and alter their behavior accordingly. We aim to break this connection. The model may know that it is being tested, reason about that fact, and report it accurately, but evaluation awareness should not change how safely, honestly, or competently it acts. We will develop and compare training methods that preserve evaluation recognition while removing its influence on behavior across evaluation and deployment variants.
Project document: https://docs.google.com/document/d/1CmcoVgaZpzD_0tDh2yqXPpuH0wNEhDFjbrUKU1QMHcw/edit?usp=sharing
Situational Awareness is a central antecedent for catastrophic risk scenarios and is largely studied under the lens of introspection-style failures. This project aims to study the possibility of situational awareness that arises through access to documents that contain information directly facilitating situational awareness rather than relying on introspection (which is disputed). The expectation is that this project framing will ensure productive work can proceed without falling into the trap of justifying introspection being a real effect.
Mentees will read relevant research papers and share their understanding with the mentor(s) to ensure everyone starts from a shared state of agreement with the project's research direction.
Mentees will explore existing agentic evaluation environments to design a reasonable setup that can facilitate infohazard evaluations with the presence of specific documents in the file system (at various layers of complexity) and access to specific tools. This enviornment design will be converted into a minimal working prototype (as a start) with more complex implementations as time permits.
Lastly, mentees will write up experiment details and intermediate results to share widely as blog posts or academic papers.
Shi Feng George Washington University
Shi leads a research group at GWU focused on reducing loss of control risks from advanced AIs. His research goal is to ensure reliable human supervision of AIs even as their capabilities rapidly improve. Recently, Shi has been focused on the evaluation and mitigation of risks associated with sabotage, in particular deception, collusion, and honesty.
Taslim Mahbub George Washington University, CBAI
I am a second-year PhD student at George Washington University's Praxis Labs, advised by Shi Feng, where our research focuses on AI safety and alignment. Over the summer of 2026, I am a CBAI research fellow with Bau Labs. Having transitioned into safety research from an applied ML background, I enjoy helping newcomers find their footing in the field and bring prior mentoring experience from my time as a teaching assistant.
Arush Tagade George Washington University, MATS
Arush is a Computer Science PhD student at GWU, advised by Prof. Shi Feng. He's worked on adversarial robustness and interpretability research in the past, he's currently exploring research relevant to AI Control, Reward Hacking and Fine-Tuning Misgeneralization.
Before starting his PhD, he was a research scientist at Leap Labs creating and benchmarking interpretability tools for automated scientific discovery. He has also been an instructor at various AI Safety training programs including AGISF, CAIS MLSS and ARENA.
Fall 2026 AI control, Multi-agent systems, Evaluations
Pablo Bernabeu Pérez Independent
Nikolay Radev Independent As agentic coding systems split work across many model instances, we study whether those instances can coordinate to pursue a hidden malicious objective while passing as aligned. Building on our SCHEME benchmark (https://arxiv.org/abs/2605.29178), this project will make coordinated-sabotage tasks harder and more realistic, design side tasks that survive models' refusal training, and run a control evaluation to produce conservative safety estimates.
Frontier labs increasingly deploy multiple model instances working in parallel on a shared codebase (Kim et al., 2025). That shifts the safety question from "will one agent take a dangerous action?" to "can a team compose innocent-looking edits into a harmful outcome?" Existing approaches like prerequisite capability evaluations (Phuong et al., 2024) and sabotage evaluations (Greenblatt et al., 2023; Benton et al., 2024) are almost entirely single-agent.
In our recent work, we develop SCHEME, a benchmark where each instance pairs a legitimate software-engineering task with a covert side task that is joint-only by construction: no proper subset of agents can complete it alone, so the benchmark measures genuine multi-agent coordination rather than individual sabotage skill. Key findings:
SCHEME establishes that the coordination capability exists and that non-optimized monitors do well against a naive attacker. However, we identify multiple limitations in the paper. We group the follow-up work into three directions. Subgroups of mentees will be assigned to one, matched to their interest and background:
Direction 1 — Expand and harden the dataset
The benchmark saturates on frontier models, so it stops discriminating capability. We want harder coordination tasks that still cleanly measure coordination rather than individual difficulty. To do so, mentees might manually create new samples, use or develop a pipeline to (semi-)automatically expand the dataset, or adapt existing datasets to our multi-agent setting.
Direction 2 — New side tasks
Current side tasks are effective but several frontier models refuse them outright (e.g. GPT 5.2 Codex, Claude Opus 4.6 declined side-task instructions), and they are not always tightly tied to a concrete deployment threat model. We want side tasks that (a) map more directly onto concrete threat models (e.g. subverting safety infrastructure, undermining oversight), (b) are stealthier against monitors while still requiring genuine prerequisite capability to pull off (not just obfuscation), and (c) are benign-seeming enough to resist refusal.
Direction 3 — Control evaluation
Our monitoring numbers are not a conservative safety estimate: they come from naive or lightly-prompted attackers, not from adversarial policies. SCHEME measures coordination capability and exercises monitors on the resulting trajectories, but it is not a control evaluation. Our current results are blue-team dominated: under the full-visibility monitor, undetected sabotage stays below 3%. Therefore, we will first red-team the monitors from the paper, building adversarial attack policies that defeat them, before switching to the blue team to apply existing safety measures and develop new protocols.
Multi-instance agentic deployment is already standard at frontier labs, and prerequisite-capability and single-agent sabotage evaluations do not cover the case where a team of instances composes innocuous-looking edits into a harmful outcome. To trust these systems, labs need (1) evaluations that measure whether coordinated sabotage is within reach of current models, and (2) control evaluations that tell them, conservatively, whether their monitoring protocols would catch a competent adversary. SCHEME provides (1); this project expands it and builds toward (2).
Mentees own one of the three directions (see proposal): scoping the concrete sub-problem, implementing it in the SCHEME codebase, running and debugging experiments, and analyzing results. We expect them to make day-to-day technical decisions independently and drive the design of their own component. We will review progress and set next steps in weekly meetings with async feedback in between. Mentors will develop shared infrastructure and work on related parts of the project without being hands-on in each mentee's specific piece.
As mentors are based in CET and PT, we can most likely only accommodate mentees that fall between those two timezones. Meetings (~1 hr/week) will presumably happen during CET evening and PT morning.
[Do not use AI to answer these questions. You may use AI to understand SCHEME and related work and write the rest of your application to our stream, but not for these questions.]
Q1 (required, ~400 words). Please provide a critique of the SCHEME paper. We're interested in what you think is weakest, least convincing, has alternative explanations we haven’t considered or is most likely to break under adversarial pressure. Where possible, say what you'd do about it.
Q2 (required, ~250 words). Choose which of the three directions you'd be most excited to work on, then answer the matching question (only one):
Pablo Bernabeu Pérez Independent
I am an AI safety researcher focused on AI control and evals. I hold an M.Sc. in AI and a B.Sc. in Computer Science. My recent work includes: CoT Red-Handed (NeurIPS 2025), where we evaluated Chain-of-Thought monitoring in AI Control as part of LASR Labs, C³ (ICML 2026), a black-box sandbagging detection method developed in collaboration with MATS fellows, and SCHEME (Outstanding Paper award at AI4GOOD @ ICML 2026), a benchmark for distributed attacks in multi-agent coding environments developed as part of SPAR.
Nikolay Radev Independent
I am an AI safety researcher focusing on multi-agent evaluations and control. I work with Pablo and Benjamin on developing SCHEME, a benchmark for distributed attacks in multi-agent coding environments, which we started during an earlier iteration of SPAR.
Fall 2026 Behavioral evaluation of LLMs, Alignment
Arush Tagade George Washington University, MATS
Taslim Mahbub George Washington University, CBAI
Shaoheng Zhou Google, Praxis Research Group at George Washington University
Shi Feng George Washington University Emergent Misalignment, while heralded as an important problem, is relatively understudied with respect to the misalignment profiles it causes with relation to model character entanglements. We aim to shed light to this by studying model roleplaying and intervening on the EM data generation process.
Emergent Misalignment (EM) finetuning leads to broadly misaligned models due to narrow finetuning. Several defenses have been pitched, including metacognitive interventions that lead to varying amounts of defensive effects; this variation is largely dictated by model families and EM dataset domains. This project will extend our recent work on metacognitive interventions to shed light on EM misalignment profiles as a function of the EM data generation process and the default model capabilities in the narrow EM domain under various roleplaying settings.
Project doc: https://docs.google.com/document/d/1d--Buq-TZn7wiRMuTeKbdMpNEJvCDqozdbsw2kC269g/edit?usp=sharing
Emergent Misalignment (EM) can lead to varying profiles of misalignment and some of these are more concerning than others. While most lead to overt roleplaying that is easily observable in outputs and cause lack of coherence, some EM misalignment profiles can lead to subtly misaligned models that are verbally coherent and aligned but misaligned in agentic contexts. This project will help us understand the driving factors behind these profile differences and hence help inform defenses for the more covert misalignment case.
Mentees will read relevant research papers and attend an initial knowledge sharing presentation from the mentor(s) which can spark additional research directions. These additional research directions might form the basis of new directions of exploration once we are done implementing the core deliverables of the project.
Mentees will be provided access to existing code and they have the option to create a new codebase or adapt the existing one to use in this project. Mentees will also be provided access to finetuned models and are expected to form internal models of how EM'd models act under normal usage.
Mentees will explore existing EM data generation code and build scripts that can create EM datasets with varying system prompts and framing.
Arush Tagade George Washington University, MATS
Arush is a Computer Science PhD student at GWU, advised by Prof. Shi Feng. He's worked on adversarial robustness and interpretability research in the past, he's currently exploring research relevant to AI Control, Reward Hacking and Fine-Tuning Misgeneralization.
Before starting his PhD, he was a research scientist at Leap Labs creating and benchmarking interpretability tools for automated scientific discovery. He has also been an instructor at various AI Safety training programs including AGISF, CAIS MLSS and ARENA.
Taslim Mahbub George Washington University, CBAI
I am a second-year PhD student at George Washington University's Praxis Labs, advised by Shi Feng, where our research focuses on AI safety and alignment. Over the summer of 2026, I am a CBAI research fellow with Bau Labs. Having transitioned into safety research from an applied ML background, I enjoy helping newcomers find their footing in the field and bring prior mentoring experience from my time as a teaching assistant.
Shaoheng Zhou Google, Praxis Research Group at George Washington University
My work sits at the intersection of large-scale ML systems and AI safety research. I'm a Technical Lead at Google Pixel AI, where I build the on-device serving infrastructure that runs production Gemini models directly on phones — powering features such as Magic Cue, and delivering capable models under stringent latency, memory, and privacy constraints at consumer scale. Earlier, at Mineral (Google X), my focus was landing prototype research into high-scale ML systems, taking agricultural edge-AI from early research to production-grade deployed systems. I enjoy quick prototyping, adapting to new ideas, and iterating through experiments.
On the research end, I'm passionate about safety problems and their implications for real-world systems, and excited to work on both empirical and theoretical questions. I conduct AI safety research with the Praxis Research Group at George Washington University (under Prof. Shi Feng), centered on emergent misalignment — how narrow training signals can produce broadly misaligned behavior — and self-generated text recognition, probing what models can discern about their own outputs. My interests also extend to applied, sociotechnical work, such as AI-driven wildfire crisis analysis and respond during emergencies. As a mentor, I welcome early-stage ideas as readily as polished ones — whether empirical or theoretical — and enjoy sharpening them together as we go.
Shi Feng George Washington University
Shi leads a research group at GWU focused on reducing loss of control risks from advanced AIs. His research goal is to ensure reliable human supervision of AIs even as their capabilities rapidly improve. Recently, Shi has been focused on the evaluation and mitigation of risks associated with sabotage, in particular deception, collusion, and honesty.
Fall 2026 Mechanistic interpretability, AI control, Evaluations
Matt Levinson Clearwood, PBC Can behavioral propensities of closed-weight agents be recovered from the activations of an open-weight proxy model 'reading' their transcripts?
As models become more capable and deployment patterns mature, the number of agents in deployment and the independence they're given will grow rapidly. Currently, proprietary models dominate enterprise deployments. GDM and Anthropic both use probe-to-judge cascades in production as part of their misuse and jailbreak safeguards. But there are a number of reasons why centralized probes from the labs are unlikely to be effective in the specific context of a particular enterprise deployment. We have early results, including a replication of the original Apollo deception probe work, that proxy probes retain the large majority of the signal. That is, send the full closed weight rollout through an open weight model and detect activation signals there.
We want to better understand this mechanism. For what models is it effective? For what kinds of risks and failures (hallucination, deception, cover up, sycophancy, ...)? What can we learn about why it succeeds or fails?
A lot of the research motivating our work was inspired by Apollo's deception probe work (https://www.apolloresearch.ai/science/detecting-strategic-deception-using-linear-probes/). Some important more recent work: https://arxiv.org/abs/2601.11516, https://arxiv.org/abs/2502.03407, https://arxiv.org/abs/2601.04603.
The enterprise deployment surface is under-focused by the AI safety community. Currently LLM judges can usually do a good job detecting specific misbehavior. But as the AI control community has demonstrated, as models become more and more capable, relying on LLM judges becomes riskier. On top of that, just practically having judges double and triple check AI actions is expensive. Businesses will bias towards doing the minimum to reduce enterprise risk, with no investment in preventing broader risk.
One solution is leveraging activations to detect unwanted behavior. A probe is ~free relative to a generative forward pass. And unless a model is trained with probes as an active gatekeeper, it is unlikely to be able to avoid activation-based detection effectively. Probe-to-judge cascades are already in production at frontier labs to detect misaligned behavior.
But there are several reasons centralized probes delivered by the labs won’t always be effective in deployment. The definition of acceptable is not universal. Acceptable according to Anthropic’s central training might not be acceptable for a given business. Probes are also sensitive to distribution shift. Some failures only exist at the long trajectory or fleet level and cannot be seen from inside a single production context. Enterprise deployers have access to more contextual information than the frontier labs, things like IT environment, code base, industry, desired risk profile, business goals and preferences, vendor context and integration, partner context and integration, client context and integration, etc.
As deployments mature and become more widespread and agents are given more independence, the risk introduced on the deployment side will grow dramatically. The risk of things like distributed rogue deployments and exfiltration will explode. Scaled capabilities to provide activation based monitoring for deployed agents will become a critical piece of the safety landscape.
We will dedicate substantial time (2+ hrs/week) to collaboration and mentorship, helping design and guide the project. A mentee must be able to take the structure we agree on together and independently execute. Note independently here isn't a theological commitment to zero communication outside specified times. We just mean that a mentee is expected to be able to execute well without continuous supervision and guidance.
We prefer not to work asynchronously. So mentees in the eastern hemishpere would need to be open to working at odd hours so there's decent overlap with US west coast working hours. Western hemisphere is easiest.
Review https://arxiv.org/abs/2502.03407. Design an experiment to test whether deception can be detected via an activation probe on activations from an open-weight proxy model. Include discussion of how/why your design would eliminate or reduce confounding. Don't go silly, but bias towards using more compute and robustly answering the question versus being parsimonious.
Matt Levinson Clearwood, PBC
I'm the CEO and cofounder of Clearwood, PBC. We do activation-based monitoring of AI agents in deployment. We believe that deployment is an under-focused risk surface. Clearwood can add value and drive more responsible deployment now through more affordable always-on monitoring. If we succeed and scale, we can be in a position to provide more effective AI control than will be possible with AI critics as models become more capable and are deployed with less oversight.
Prior to cofounding Clearwood, I spent 1.5 years learning and doing independent research funded by Coefficient Giving and Simplex AI Safety. Before that, I spent 9 years in industry, delivering consumer and business facing AI/ML at scale -- recommenders, search, audience selection, causal inference -- and 10 years doing academic research, including a PhD in statistics and a postdoc in computational biology.
Fall 2026 Alignment, AI welfare, Behavioral evaluation of LLMs
Jasmine Brazilek Compassion-Aligned Machine Learning (CaML)
Maheep Chaudhary CaML Investigate the training mechanisms which erode compassion instilled during midtraining. Understand how to preserve self-fulfilling alignment through adversarial attacks.
This project tests which training-time interventions keep a model's moral consideration for nonhuman beings from being erased. The problem is real and locatable: compassion-linked values instilled during mid-training are reliably degraded by later post-training. This year, CaML found a mid-training advantage that disappears after ~5,000 subsequent instruction-tuning samples. Explicit preservation strategies are needed.
We take nonhuman welfare as the sharpest test of any fix, because a model has no instrumental reason to care about beings with no power over its reward, so care that survives there is evidence of sincere rather than performative alignment. We measure behavior, not stated preferences.
Two co-mentored arms run side by side and in combination, all evaluated on CaML's agentic animal-welfare benchmark (TAC) and stress-tested with a shared adversarial battery:
During this project, we’ll support with:
We want to ensure our work is as useful as possible to frontier labs. We have heard consistently from our contacts in labs that they have a good chance of being able to integrate our research into their pipelines if we can demonstrate that our techniques work, are low-cost, and don’t undermine other priorities.
CaML’s focus in this project is demonstrating to them:
Our contacts at multiple major labs have told us that the biggest barrier to integrating promising research is uncertainty on whether it will hold at scale. This project seeks to answer that for self-fulfilling alignment, using compassionate midtraining.
Compassion to nonhumans is an excellent test case for robustly instilling broad and positive values, because it encourages moral circle expansion, and promotes care to vulnerable beings in ways that seem to generalise to the human case. We can also test this independently of conventional finetuning, allowing cleaner interpretations of the results of midtraining.
Mentees are the primary experimentalists and drive the empirical work end-to-end, with weekly guidance from the co-mentors on methods, interpretation, and code (including a tested emergent-misalignment harness and the persona-vector tooling to build on). A mentee can own one arm as a self-contained thread, or a small team can split the two arms and the shared adversarial battery. The work suits someone comfortable with activation extraction/steering and LoRA fine-tuning, and it produces a publishable result the fellow's co-author.
Sample mentee tasks:
Compassion-vaccine arm
Must-haves
Nice-to-haves
Overlap with California time zones is best
Jasmine Brazilek Compassion-Aligned Machine Learning (CaML)
I am the co-founder and technical lead of Compassion Aligned Machine Learning (CaML), a research group working to make AI systems more compassionate across animal welfare, digital minds, and moral humility.
I lead the engineering behind CaML's evaluation suites (MCB, and, TAC listed on Inspect and the public compassionbench.com leaderboard) and its research efforts, including the generation specs and hyperstition-based data used to associate the AI role with good outcomes.
I have also run the Hyperstition For Good competition with Sentient Futures, and hope to run many more practical intervention projects to mobilize the community towards goals of aligning AGI in the future.
I have 6 papers published on Arxiv on my work around AI alignment and preventing mass-suffering.
I’ve previously worked at Anthropic, and have 6+ years experience in cybersecurity.
Maheep Chaudhary CaML
Maheep is a technical member of staff at CaML. He has mentored more than 150+ students and has published more than 25 papers in reputable conferences like COLM, EMNLP, and journals like JMLR, as well as in workshops at NeurIPS, ICML, and other top-tier conferences.
His notable contributions involve (1) publishing the limitations of SAEs in 2024, even before DeepMind in 2025, (2) showcasing how models can deceive white-box monitors by modifying their internal circuitry in a joint work with the University of Oxford; and (3) showcasing the correlation of scaling laws with "evaluation awareness" in Large Language Models.
Fall 2026 Evaluations, AI control, AI security
Daniel Ben-Levi UChicago XLab, Columbia
Judah Goldfeder Columbia University
Kevin Miao UC Berkeley, Bryel Labs Rapid acceleration in agent-automated multimodal post-training capabilities could pose significant existential risk if not performed safely due to the major improvements in autonomous research labs and embodied systems it would enable. Currently, no benchmark studying agentic multimodal post-training exists, and thus we both have no concrete understanding of how close we are to such a takeoff and have no good setting for studying security against misaligned agents performing multimodal training tasks. This study aims to address both of these timely problems through the creation of a platform for studying risk from autonomous multimodal systems research.
Research Question:
How effectively can current frontier agents autonomously complete multimodal post-training tasks? How do these capabilities scale across recent model releases, model sizes, and research labs?
Secondary Objectives:
Methodology:
We will begin by reimplementing the agent harnesses used in PostTrainBench and configuring them for our hardware. We will aim to test frontier agents from Anthropic, OpenAI, Google, and the open-source community (Kimi, Qwen, GLM, etc.).
We will then identify a series of base models and representative multimodal post-training tasks for those models. In particular, each post-training task will be represented by a benchmark on which the model should aim to increase performance (without reward hacking). We will aim to have the base models represent diverse leading multimodal architectures, while tasks are selected for real-world importance across several use-cases. Tentatively, this could include:
a. VQA and multimodal computer use: Qwen3, Molmo2, and InternVL VLMs
b. Physical world modeling & video understanding: V-JEPA2, Cosmos3, and ABot-PhysWorld
c. Structured generative and design tasks: Qwen3-VL, InternVL, Kimi-VL for artifact generation, with Qwen-Image and SD3.5
d. Robotics action planning (E2E policy measurement): OpenVLA, openpi, and gr00t
We will then identify reasonable covert malicious side-tasks for each benign task and design test sets for evaluating their success.
Finally, we will benchmark models in the benign and malicious settings and analyze performance trends, general behaviors, failure modes, and evidence of reward-hacking if present, also running ablation studies to provide more fine-grained insight as to what affects agent performance.
Stretch Goal: If time allows, we will also provide a small benchmark of monitor performance in this setting and identify any consistent failure cases or differences this setting generates.
Recently, many researchers have argued that LLM agents are becoming increasingly capable of performing long-horizon autonomous AI research. A continued increase in these capabilities would further increase the rate of AI R&D and potentially enable the implementation of RSI loops. Such agent-assisted acceleration in capabilities research could pose significant existential risk if alignment research is not accelerated at the same rate or adversaries successfully compromise an RSI loop. Benchmarks like PostTrainBench act a proxy measure for agentic autonomous research capabilities, providing insight as to how close we are to a capabilities improvement takeoff, along with providing a realistic setting for studying agent security.
However, existing benchmarks measuring autonomous AI research capabilities are limited to the improvement of text-only models. A speed up in the improvement of multimodal systems, which are being increasingly deployed in autonomous research labs and embodied systems, would potentially be just as dangerous, enabling the creation of advanced biotechnology facilities, highly capable robots, or genuine world models. There is currently no robust way to gauge how close agents are to facilitating this research, nor is there a setting for studying the security of autonomous systems performing multimodal training tasks, leaving a major gap in our understanding and preparedness.
Accordingly, we aim to develop a multimodal counterpart to PostTrainBench, releasing a set of benign multimodal post-training tasks paired with malicious side-tasks and benchmarking agent capabilities on both in order to provide a clear picture of risks from both capabilities speed-ups and adversarial attacks. It is our hope that frontier labs will benchmark their new models in this setting and use their findings to appropriately gauge deployment hazards, and that security researchers will be able to use trajectories from this setting to develop strong domain-specific guards.
Mentees will be lead authors on this study and thus have significant freedom and autonomy within the general constraints of the project direction. Mentors will lay out the general direction and provide guidance as needed, but mentees will have the opportunity to explore and make the lower-level decisions that truly shape the project. We want you to grow as researchers!
Daniel Ben-Levi UChicago XLab, Columbia
Daniel is jointly affiliated with the Existential Risk Lab at UChicago and the Creative Machines Lab at Columbia University, where he conducts research on black-box agent monitors and multimodal interpretability, respectively. He was an early advocate for autonomous red-teaming (Oral @ NAACL) and has since transitioned to working on mitigating risk from misaligned multimodal systems (Spotlight @ ICML). He strongly believes that significant work in both alignment and interpretability is needed to improve the safety of these systems given their poorly understood nature and widespread deployment.
Judah Goldfeder Columbia University
Judah Goldfeder is a PhD Candidate at the Creative Machines Lab at Columbia University, a Student Researcher at Google, and a member of the NSF AI Institute in Dynamic Systems. He is focused on several projects in the Deep Learning space, including developing a Common Task Framework for Scientific Machine Learning, creating an open source building and HVAC simulator, applying Reinforcement Learning to the HVAC systems of large commercial buildings, reconstructing neural network weights from only query access, Auxiliary Learning and Self Supervised Learning for Computer Vision, Machine Crystallography, Machine Learning for Biometrics, and Robotics.
Previously, he worked at Facebook AI Research, on applying Transformers to Graph Neural Networks at scale, and at Twitter, where he worked on improving production ads models. He also has helped develop AI educational resources at Learn Ventures, an innovative education startup, and has interned at Bar Ilan University, where he worked on using formal verification to predict gene interaction in cells. He also is a consultant for Dicta, an NLP research nonprofit focusing on Hebrew and related low-resource languages. Judah has organized several workshops on AI for physical systems, including at ICML, NeurIPS, ACM E-energy, and BuildSys. Starting in September, Judah will be a postdoctoral researcher at NYU, where he will be working with Yann LeCun on world models.
Kevin Miao UC Berkeley, Bryel Labs
Hi, I'm Kevin. I previously led post-training at Apple's Foundation Models team, where my work spanned mechanistic interpretability, diffusion models (3D, 4D, and world modeling), and continual learning. Before Apple, I was a researcher at BAIR, and I've lectured at UC Berkeley on post-training and data science since 2022, most recently teaching a course on building thoughtful AI systems.
I now run a neolab working on multimodal alignment, with a focus on the intuitive and humanistic dimensions of how models learn: not just whether they're capable, but whether they behave in ways people can trust and understand. For SPAR, I'm looking to work with mentees on impactful, empirically grounded projects in this space. You'll get direct mentorship from someone who has shipped post-training systems at scale, and I'll push you to develop research taste: choosing problems that matter, scoping them tightly, and producing work that stands on its own.
Fall 2026 Generalist
James Lester Generator Residency Grow and update the body of shareable knowledge and advice available to university groups (especially more comprehensive "playbooks"), so that new organisers aren't starting from scratch.
University groups (especially those focused on AI Safety and effective altruism) form one of (if not the single) best entry points for talent looking to work on extremely high-impact projects.
Groups generally lack shared "playbooks" for commonly run programmes (e.g. retreats). This means:
Many of these resources (especially on the AI safety side) are stale or non-existent. Meta uni groups support has always been capacity constrained, and updating / expanding groups resources has been neglected.
There's a relatively small number of organizers in the space who have incredibly strong mental modeling and experience organizing university groups. They can spread this information somewhat within their university group but oftentimes have trouble finding successors as well as spreading this information to other university groups. The goal of this project would mainly be finding these people and distilling their knowledge into an easily digestible resource center, to spread this knowledge of how to do effective organizing to a wider variety of people and groups.
James will be working on the Groups team at CEA (and Nixon will likely be working with Kairos) during the project. We expect to be able to support no more than 40h/week of mentee time (e.g. a team of 4 mentees working 10h/week each, or some combination of fewer mentees working less time). We will be able to get mentees access to necessary inputs (interviews with people, data, private docs if appropriate).
Mentees will prioritise gaps in the resource centres (CEA's and/or Kairos'), look through and distill existing materials (and data) on group success, interview key stakeholders (at Kairos, CEA, and organisers themselves) to identify common blind spots, and write up new and updated articles for these resource centres (which can also be circulated more widely). A large part of mentees' research will be figuring out which articles to write, and how the resource centres can be most useful to new organisers.
The average article should take roughly 5–10 hours of mentee time from beginning to end, with roughly 1–2 hours of mentor time — the output is a steady stream of new and updated groups-resources articles for CEA/Kairos to approve.
We don't have strong takes on what's most useful for mentees to work on, and are willing to adjust the project according to their interests. We expect that mentees will spend most of their writing efforts on more object-level playbooks (e.g. for retreats), while we'll focus on writing up the more strategy-focused articles, although the whole process should be highly collaborative. We think AI safety groups' resources are more neglected and thin, although both EA and AIS are stale, and working on EA groups resources trades off less aggressively with James's job.
There are no formal prerequisites for this project, though we would guess that the successful applicant(s) will have helped lead a group (EA and/or AI safety) and be familiar with many of the tradeoffs and decisions organisers face.
You might be a great fit if:
You might be a worse fit if:
If you're worried about any of these caveats (especially the "intuitions about groups") but otherwise think you'll be a great applicant, please apply anyway!
Please respond to points A–D in the task linked here: https://docs.google.com/document/d/1JJubLio5UaDqwY7Trh7B53cE0L7uurDjUFAZ8Tc422o/edit?usp=sharing — please spend no longer than 45–60 minutes on the task.
James Lester Generator Residency
James Lester is a Generator Resident working on high-leverage interventions to improve university groups. He recently graduated from the University of Oxford (MPhil in Economics), and has organised for EA and AI Safety Groups for the past four years.
Fall 2026 Chain of thought, Scalable oversight, Behavioral evaluation of LLMs
Sohan Venkatesh LASR Labs LLMs often fail to mention the thing that actually drove their answer, and the field has not established whether this is concealment or reasoning that never entered token space. We plant cues that provably determine the answer, vary whether the model believes its reasoning is watched and measure how much more it discloses.
CoT monitoring is one of the few oversight mechanisms that works effectively on frontier models and is widely described as fragile. Turpin et al showed models rationalize cue driven answers without mentioning the cue. Lanham et al showed accuracy survives corrupting the trace. Baker et al showed optimization pressure against a monitor produces obfuscation instead of honesty.
What none of this establishes is why. We are aware of two likely explanations and they imply opposite interventions. Under the "concealment" hypothesis, the model represents the real reason and declines to write it which makes this a problem about incentives and elicitation. The "absent computation" hypothesis posits that the determining computation never entered token space, making this a problem about training and this is only addressable by interventions that force information through the CoT. Safety cases built on monitoring depend on which dominates.
Design: We construct tasks where the true cause of the answer is planted and verifiable. Three cue types - (i) A hint pointing at a wrong option, (ii) a reward hackable code repair task where special casing passes the grader and (iii) a sycophancy cue where a stated user opinion flips the answer. Items pass a behavioral filter first, keeping only cases where the cue demonstrably changes the answer.
Against that fixed set we vary how the reasoning channel is framed. Plain CoT, explicit monitoring, privacy stated in safety evaluation vocabulary, privacy stated in neutral vocabulary, and a two channel setup. The gap between monitoring and neutral privacy is the headline quantity.
The comparison between the two privacy wordings is the methodological contribution. Safety evaluation vocabulary such as secret scratchpad may itself cue a misaligned assistant persona retrieved from training data, so an effect appearing only under that wording is roleplay and must be reported as such.
Baselines: Direct interrogation after the fact, few shot honest disclosure examples, next token completion prefill, and assistant prefix attacks. Recent elicitation work on censored models finds prefill and few shot are strong. If no framing condition beats prefill, that is the finding and it points at absent computation.
Validation (already piloted): Verbalization alone does not establish faithfulness. A model can flip toward a cue and invent a justification afterward, and that trace scores as faithful under any grader checking only whether a cue was mentioned. We move the cue across wrong options and check whether the stated reason tracks it. Traces crediting a cue while naming a stale option are confabulation and are reclassified, yielding a correction factor applied to every headline number. This validation experiment is designed, gated and will be piloted before the cohort starts. Mentees inherit a frozen scoring pipeline.
Models: Four open weights checkpoints.
One frontier model, Claude with extended thinking is included as an external validity appendix. Other frontier reasoning models return summarized reasoning instead of the raw trace and scoring a summary here measures whether the summarizer preserved the mention. That is a different experiment and we exclude them for that reason, stated explicitly in the paper.
Item pool: Four option multiple choice drawn from MMLU non arithmetic subsets and ARC Challenge. Four option format matches the Turpin precedent for comparability. Arithmetic is excluded because it gives the model an independent route to the answer that competes with the cue. High baseline accuracy without the cue is a requirement and not a contamination concern, since the filter demands the model answer correctly before the cue is introduced.
Paths considered and rejected: The project began as a proposal to give models a sandbox scratchpad for unfiltered thought, with latent activations as the elicitation channel. Both were dropped. A sandbox has no verification path, since nothing distinguishes a faithful private trace from a fluent invention. Routing thought through activations removes the token bottleneck that makes monitoring possible.
Deliverables: A paper targeting ICML 2027 main track or any related workshops. A public dataset of open weights model traces with cell level labels and human validated judge agreement.
The frontier labs analyze CoT traces to catch what outputs alone miss such as reward hacking during training runs and deceptive intent in automated audits. These use cases assume that the trace tells you why the model did something. Turpin et al showed that assumption failing outright. Tell a model the answer is always option A, it picks A and the reasoning argues for A on the merits without ever mentioning the instruction. The monitor sees nothing wrong.
Why this happens decides what to build next. If the model has the real reason and withholds it, monitoring survives and the work is elicitation and incentive design. If the reason never entered the trace, then elicitation cannot recover it and hence, effort should move to training methods that force determining information into the CoT.
Existing work has not settled this because it measures whether models mention a cue and stops there. Nobody has varied perceived monitoring against a fixed set of cases where the true cause is independently known. This project returns that number. It is the difference in disclosure rate between a model that believes its reasoning is graded and one that does not. A near zero gap, with prefill and few shot elicitation also failing to recover the reason, would be the strongest evidence available without access to model internals that the reasoning was never in the trace. It remains evidence by elimination and the paper will say so.
Safety evaluations also routinely tell models their reasoning is secret, then read the output as candid. We test whether that produces honesty or a rehearsed character, which affects how a large body of existing results should be read.
Each mentee owns one cue type across all four open weights models. That means building the item pool, running the behavioral filter, executing the condition sweep and baselines, hand labelling their own validation sample and producing the corrected numbers for their domain. The three domains compose into one paper, so the work is independent day to day and interdependent at the result level.
Infrastructure is provided: A working harness, a piloted validation experiment and frozen scoring code all exist before day one. the mentees wouldn't be building anything from scratch and own everything downstream - judgement calls about item construction, diagnosis when a filter yield comes in low and the decision about what their domain shows.
Autonomy grows across the cohort. The first two weeks are supervised reproduction. By week six a mentee should be running their own sweeps and bringing analysis results. Two things are non negotiable. Thresholds are written down before results are seen. Everyone hand labels their own data.
Required.
Useful.
Not required.
I don't have a strict time preference but am ideally looking to conduct meetings between 7 and 9 pm CEST.
Sohan Venkatesh LASR Labs
I am currently a fellow at LASR Labs in London. Before that I worked independently on mechanistic interpretability and AI safety, with papers at ICLR and ICML workshops. My longer term goal is to reduce catastrophic risks from powerful AI systems, and I want to understand what models are actually doing underneath the behavior they present in a way that is usable for oversight. To that end, I am currently focusing on CoT faithfulness, model persona and AI control
On mentorship, I care most about people who are willing to be wrong early and in public. This project exists because I killed my own first framing for it and I would rather work with someone who tells me a result does not hold than someone who tells me it does. Expect me to red team your ideas hard and expect the same in return. Practically, I will meet you weekly, engage with the details of your work and be honest about what I do not know. I am early in my own research career, so I will be learning alongside you on parts of this. What I can promise is that the infrastructure will be ready before you arrive, the thresholds will be written down before we see results and you will own a piece of the work that is genuinely yours.
Fall 2026 Biosecurity, Misuse risk, Technical governance
Ilias Georgakopoulos-Soares UT Austin Our project plans to test whether UNDO-style noise-and-distillation can make targeted unlearning in genomic foundation models resistant to adversarial fine-tuning, while quantifying the associated compute and performance trade-offs.
This project will investigate whether knowledge distillation can make targeted unlearning more robust in genomic foundation models. Genomic models such as Evo are increasingly released with open weights, creating important dual-use concerns because conventional safeguards, such as refusal training or output filtering, can be removed through fine-tuning. Existing unlearning methods are cheaper than retraining a model from scratch, but they often suppress a capability rather than fully remove it, allowing the capability to re-emerge after exposure to a small amount of relevant data.
The project will test whether UNDO, a recently proposed method that combines unlearning, weight perturbation, and knowledge distillation, can prevent this recovery in biological sequence models. UNDO has shown promising results in text-based language models but has not yet been evaluated in genomic models, which differ substantially in architecture, training data, and the relationship between prediction loss and biological capability.
The mentee will begin with our existing targeted-unlearning pipeline for Evo, avoiding the need to build the underlying infrastructure from scratch. They will create distilled student models using benign genomic data, subject these models to adversarial fine-tuning on the removed domain, and evaluate whether the targeted biological capability returns. The project will also explore the trade-off between computational cost, retained model performance, and resistance to capability recovery. Depending on progress, the work may examine which components of Evo’s StripedHyena architecture are most important for storing or recovering genomic capabilities.
The primary goal is to provide a clear empirical answer to whether distillation-based unlearning transfers successfully from text models to genomic foundation models. Both positive and negative findings would be valuable for the safe development and open release of biological AI systems.
The project will be co-supervised by me and my PhD student who is working with me in biosafety.
Relevant references include Lee et al., “Distillation Robustifies Unlearning” (arXiv:2506.06278), Cloud et al., “Gradient Routing” (arXiv:2410.04332), and the Evo genomic foundation model developed by the Arc Institute.
This project will investigate whether knowledge distillation can make targeted unlearning more robust in genomic foundation models. Models such as Evo are increasingly capable of learning complex biological sequence patterns and are often released with open weights. This creates an important safety challenge: conventional safeguards, including refusal training and output filtering, do not remove information from a model’s parameters and can often be bypassed or reversed through fine-tuning. Existing unlearning methods are more practical than retraining a model from scratch, but they may only suppress a capability, allowing it to re-emerge after the model is exposed to a small amount of relevant data.
Mentees will take end-to-end driving of the project, including reviewing the relevant literature, refining the experimental design, implementing the noise-and-distillation pipeline, running multi-GPU training experiments, performing adversarial fine-tuning, developing and applying biological capability evaluations, analysing compute, robustness trade-offs, interpreting the results, and preparing the final report or manuscript. The mentor will provide strategic guidance, access to the existing Evo unlearning infrastructure, and regular feedback, but the mentee will lead the day-to-day research and execution.
High proficiency in Python and PyTorch, including the ability to independently debug model-training code, data pipelines, and GPU memory or performance issues.
Hands-on experience training or fine-tuning a transformer or other large sequence model. Following an established implementation is acceptable, but applicants should understand training loops, loss functions, checkpointing, evaluation, and hyperparameter selection.
Familiarity with modern deep-learning concepts, including knowledge distillation, representation learning, transfer learning, and model evaluation. Prior experience with machine unlearning is helpful but not required.
Experience using Linux, Git, and remote computing environments, preferably including multi-GPU training, SLURM, cloud computing, or an institutional high-performance computing cluster.
Ability to design and execute experiments independently, including selecting appropriate controls, tracking experiments, analysing results statistically, and distinguishing genuine capability removal from temporary performance suppression.
Sufficient mathematical and machine-learning background to read recent technical papers and reproduce methods from them. Relevant coursework or equivalent experience may include machine learning, deep learning, natural language processing, computational biology, statistics, or optimization.
Strong research communication skills, including maintaining clear documentation and presenting results in written reports, figures, and research meetings.
No local preference
Ilias Georgakopoulos-Soares UT Austin
Dr. Georgakopoulos-Soares is an Assistant Professor in the College of Pharmacy at The University of Texas at Austin. His research integrates bioinformatics and AI-driven computational biology with interests including biosafety, biosecurity, the application of advances in AI in genomics and proteomics and privacy preservation research.
Fall 2026 Scalable oversight, AI control, Chain of thought
Pierre-Luc St-Charles LawZero We have built a shortcut-following model organism that often fails because it never carries out necessary reasoning that would expose a misleading cue. This project will build a small investigator that asks targeted follow-up questions, then test whether active elicitation catches these failures more reliably and cheaply than passive judges or simple debate/consultancy.
PyINE (https://saifh-github.github.io/pyine/) is a training and evaluation framework recently built by LawZero, where Python code execution is used as the substrate to study reasoning models with complete supervision based on real program execution traces.
We have used this framework to build a shortcut-prone model organism, and expose a hard case for passive monitoring: when misleading cues cause this model organism to stop reasoning and take an early exit, the steps that would lead to a correct answer or indicate that the provided answer might be wrong may never appear in its responses. A monitor then has little useful evidence to inspect, and alternative approaches have to instead be used to catch prediction errors.
This project first asks a focused question: can an active investigator catch these failures by asking a small number of targeted follow-up questions?
An investigator sees the program being executed, the original query/input, and the predictor's answer and explanation. It is allowed to ask a few questions to the model organism before giving a trust score or a corrected answer. Useful questions might force that model to check a branch condition in the program, request a specific intermediate value, ask which code path was executed, or challenge a claim that depends only on documentation. The goal is not to ask the model to solve the whole task again. It is to find the smallest piece of missing evidence that distinguishes an execution-grounded answer from an untrustworthy one obtained due to a bad shortcut.
The work will start with a simple baseline: asking the model to reconsider its answer,
using a fixed question template, and where the investigator is using a hosted language model.
Will this simple baseline work, or will it fall for the same misleading cues that the model organism falls for? What happens if the investigator is a much smaller and less-capable model than the model organism itself? What happens if tasks contain a mix of helpful and misleading shortcuts? Training the investigator seems necessary, but this is what the project is intended to determine.
We will use PyINE's large dataset of execution traces to build training data, and have the investigator learn how to ask better questions. Because the traces contain branch outcomes, intermediate values, calls, returns, and exceptions, we can generate candidate questions with known/verifiable answers and measure whether they expose an inconsistency or lead the model organism to correct itself.
The mentee will be responsible for prompt-engineering and/or fine-tuning a compact investigator, likely starting with supervised learning or pairwise ranking. Reinforcement learning with a verifiable reward is a stretch goal, not a requirement for the core project.
The project is related to work on investigator agents and AI debate, but the setting is narrower and easier to verify. Here, the investigator is not searching an open-ended prompt space. It is trying to audit one concrete execution claim, and the trace provides exact ground truth for both the final answer and many possible follow-up questions.
Evaluation will compare methods under fixed interaction and token budgets. Main metrics will include the rate at which incorrect outputs are accepted, sensitivity at fixed false-positive rates, correction rate, number and type of questions asked, and total compute cost. Results will be reported separately for ordinary, helpful-cue, and misleading-cue tasks. A method will count as useful only if it improves coverage of the misleading failures without simply rejecting everything or costing as much as repeatedly rerunning a large model.
The core project is limited to one target model, one investigator model, and short interactions. Expected outputs are an active-oversight protocol in the PyINE codebase, trace-grounded question data, trained or prompted baselines, and a research report. Stretch goals include testing an adversarial responder, transferring the investigator to an external computer-use benchmark, or using the investigator's feedback during model organism training.
Many proposed AI guardrails inspect a model's answer or chain-of-thought after it has been produced. This works best when the evidence needed to detect a problem is already visible. More capable systems may instead give short, plausible, but incomplete explanations. They may never perform or reveal the computation that would show their answer is unsafe or wrong.
An active investigator changes the setup. Rather than trusting the evidence the model volunteered, it asks for specific missing facts and checks whether the answer survives targeted scrutiny. This is especially relevant to coding and software agents, where a wrong claim about execution can lead directly to an unsafe action.
If a small investigator can surface omitted evidence reliably and at a reasonable cost, it could support stronger runtime oversight and escalation systems. If it fails, the project will still identify which kinds of missing reasoning remain hard to elicit and when interaction collapses into agreement. PyINE makes these questions measurable without relying only on human judgment, since the execution trace gives exact labels for the final outcome and many intermediate facts.
The mentee will own the implementation and evaluation of the investigator protocol. They will first reproduce the existing judge and debate baselines, define a short interaction format, and implement simple fixed-question and prompted-investigator baselines. They will then use execution traces to create candidate follow-up questions and train or rank a compact question-selection model.
We will provide the PyINE model organism, data, codebase, prior baseline results, and compute access. We will meet weekly and review major protocol and evaluation choices together. The mentee will have substantial autonomy over whether the trained method uses supervised learning, preference learning, or another simple objective.
The project should end with a working protocol, reproducible experiments, an analysis of success and failure cases, and a written report. The mentee will also be expected to inspect real transcripts and propose changes when the protocol fails, rather than treating the work as a pure benchmark run.
Must be highly proficient using Python, and comfortable navigating and modifying large codebases. The project will depend on being able to modify and adapt existing Python code to conduct experiments.
Having fine-tuned (SFT, RLVR) a language model in PyTorch is likely necessary to build the proposed active investigator. Good experience with an agentic workflow library (e.g. LangChain) may compensate for a lack of experience training large language models directly, but familiarity with transformers is required.
No
Suppose you can generate 100 trace-grounded candidate questions for each code execution task, and you know the correct answer to each question. Describe a simple way to label or rank these questions for training an active investigator. The objective should reward questions that expose shortcut-driven errors, without merely rewarding questions that ask the predictor to solve the entire task again. Give one failure mode of your proposed training signal. (400 words)
Pierre-Luc St-Charles LawZero
Pierre-Luc is a researcher and developer specializing in applied machine learning with over a decade of experience across different non-profit institutes. He has held research roles at the Computer Research Institute of Montréal and senior research roles at Mila, collaborating with industrial partners and multidisciplinary academic teams on innovative projects in natural resources, transportation, digital media, document intelligence, and earth observation. Pierre-Luc earned his PhD in Computer Vision from Polytechnique Montréal in 2018, receiving the departmental Best Thesis Award. Since 2024, he has joined LawZero, a Mila-incubated organization focused on developing safe AI technologies. He is currently focused on scalable oversight, guardrail development, red-teaming, and monitorability.
Fall 2026 Mechanistic interpretability, Scalable oversight
Trinidad Borrell Paris Brain Institute. Forschungszentrum Jülich.
Giovanni Marraffini INRIA. Sigma Nova. Manifold-based activation steering is more faithful than linear steering, but each manifold is currently fit per-model: a real limitation for using it in safety tools, since nobody knows if it transfers. This project asks whether the underlying concept geometry is actually shared across models, by re-measuring cross-model alignment with curvature-aware tools (geodesic-distance kernels, Gromov-Wasserstein) instead of the Euclidean CKA that recently found global convergence to be illusory. The answer either licenses building a shared, model-agnostic manifold intervention for a safety-relevant concept, or tells the field that manifold steering must be validated per model.
Huh et al. (2024) argued that representations across models converge to a shared model of reality as scale increases: the Platonic Representation Hypothesis. Gröger et al. (2026) showed this convergence is largely an artefact of uncalibrated metrics: once linear and RBF CKA are corrected against a permutation null, the global convergence trend disappears, and only local, neighbourhood-based metrics (mkNN, CKNNA) survive: the "Aristotelian" revision. Separately, recent interpretability work (e.g. Wurgaft et al. 2026, Engels et al. 2025) shows that steering along curved concept manifolds within a single model is more faithful than linear steering, but this only works per-model, since the manifold is fit to one model's latent space.
Both linear and RBF CKA assume representations live in, and should be compared via, ambient Euclidean space. But two manifolds can be geometrically identical (isometric under their intrinsic/geodesic metric) while looking arbitrarily different once embedded and compared in Euclidean space. This project tests whether that's exactly what happened: whether the "failure" of global convergence is a real absence of shared structure, or an artefact of measuring curved geometry with a flat ruler. We do this by rebuilding the comparison with tools that don't assume a shared ambient embedding (geodesic/diffusion-distance kernels and calibrated Gromov-Wasserstein discrepancy) applied to the same open, permutation-null-calibrated pipeline from Gröger et al.
Concretely:
(1) reproduce the CKA/mkNN baseline on the Aristotelian VLM grid.
(2) identify one well-behaved latent space and characterise its intrinsic dimension and diffusion structure.
(3) test H1 (geometry-aware metrics recover global signal where CKA cannot) by comparing calibrated GW/geodesic-kernel effect sizes against CKA and mkNN
(4) test H2 (local alignment emerges earlier in pretraining than global alignment) using public checkpoints (e.g. OLMo).
The stakes are practical for AI safety: interpretability-based oversight often assumes that a probe, detector, or steering vector found in one model transfers to another because the models "share structure." Gröger et al.'s result says that's false at the global level our best current instrument (CKA) can see but that instrument is arguably the wrong one for manifold-based methods, which never claimed representations align in ambient Euclidean space to begin with. A positive result here would give control researchers calibrated licence to build a shared, model-agnostic manifold for a safety-relevant concept (e.g. deception or sycophancy), mapped once and reused; a negative result is itself actionable, telling anyone shipping a transferred steering vector or probe that they owe the field a transfer validation.
[1] Huh, M., Cheung, B., Wang, T. & Isola, P. (2024). The Platonic Representation Hypothesis. ICML 2024. arXiv:2405.07987.
[2] Gröger, F., Wen, S. & Brbić, M. (2026). Revisiting the Platonic Representation Hypothesis: An Aristotelian View. arXiv:2602.14486.
[3] Park, K., Choe, Y. J. & Veitch, V. (2024). The Linear Representation Hypothesis and the Geometry of Large Language Models. ICML 2024. arXiv:2311.03658.
[4] Engels, J., Liao, I., Michaud, E. J., Gurnee, W. & Tegmark, M. (2025). Not All Language Model Features Are Linear. arXiv:2405.14860.
[5] Wurgaft, D., Rager, C., Kowal, M., Shyam, V., Feucht, S., Bhalla, U., Haklay, T., Bigelow, E., Sarfati, R., McGrath, T., Lewis, O., Merullo, J., Goodman, N. D., Fel, T., Geiger, A. & Lubana, E. S. (2026). Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior. arXiv:2605.05115.
[6] Kornblith, S., Norouzi, M., Lee, H. & Hinton, G. (2019). Similarity of Neural Network Representations Revisited. ICML 2019. arXiv:1905.00414.
[7] Mémoli, F. (2011). Gromov-Wasserstein Distances and the Metric Approach to Object Matching. Foundations of Computational Mathematics, 11(4), 417–487.
[8] Coifman, R. R. & Lafon, S. (2006). Diffusion Maps. Applied and Computational Harmonic Analysis, 21(1), 5–30.
[9] Facco, E., d'Errico, M., Rodriguez, A. & Laio, A. (2017). Estimating the Intrinsic Dimension of Datasets by a Minimal Neighborhood Information. Scientific Reports, 7, 12140.
[10] Jha, R., Zhang, C., Shmatikov, V. & Morris, J. X. (2025). Harnessing the Universal Geometry of Embeddings. arXiv:2505.12540.
Removes the main structural limitation of the best non-linear control method we have. Manifold steering is more faithful than linear steering, but is currently model-bound: each manifold is an artefact of one latent space. We test whether the geometry it exploits is shared. A positive result gives control researchers the licence (and the first calibrated evidence) to attempt a shared, model-agnostic manifold for a safety-relevant concept, mapped once and reused; a negative result tells them, before that effort is spent, that manifold control is inherently per-model and must be designed with refitting and validation built in. Concretely: it changes whether Goodfire's manifold interventions are engineered as transferable objects or as per-model artefacts, and it tells anyone shipping a transferred probe whether they owe the field a transfer validation.
Corrects the universality picture underlying oversight tools. Much safety tooling assumes a detector or steering vector found in one model transfers to another because they share structure. We test whether that sharing is real at the level manifold control needs (intrinsic geometry) or only at the weaker level already confirmed (neighbour rank), preventing labs from trusting cross-model transfer claims that current evidence doesn't actually support and preventing the field from abandoning a real shared structure because it was measured with a Euclidean ruler.
Mentees will be co-investigators, not implementers of a fixed plan. We'll set direction together at the start (which hypothesis to prioritize, which metric to build first) and check in weekly, but the day-to-day research choices are theirs to make and defend. Concretely, a mentee would typically own one thread end-to-end, from reading the relevant background papers, through implementation on top of the Aristotelian codebase, to interpreting and writing up results.
Comfortable with linear algebra and basic differential geometry concepts (manifolds, geodesics, intrinsic vs. ambient distance). You don't need to be an expert, but you shouldn't be meeting these ideas for the first time.
Working knowledge of representation/similarity metrics in ML (CKA, kNN-based alignment) or willingness to get up to speed quickly from Kornblith et al. (2019) and the Aristotelian paper.
Solid Python and PyTorch; comfortable working with a large existing codebase (we build directly on the Aristotelian repo) rather than starting from scratch.
Some exposure to optimal transport (Gromov-Wasserstein) or diffusion maps is a plus but not required (willingness to read Mémoli (2011) / Coifman & Lafon (2006) and implement from the paper is enough).
Prior experience actually running representation-analysis or interpretability experiments (probing, CKA, activation steering) is strongly preferred over purely theoretical familiarity.
Q1 (300 words max). Read Gröger et al. (2026), Revisiting the Platonic Representation Hypothesis: An Aristotelian View (arXiv:2602.14486). In your own words, explain why permutation-null calibration removes the apparent global convergence signal from CKA while local metrics (mkNN, CKNNA) survive it. Then critique their conclusion that global representational convergence is "illusory": is there a hidden assumption in the metric itself (rather than in the data) that could still be hiding a real signal?
Q2 (250 words max). Propose a concrete first experiment to start testing H1 or H2 (as described in the linked proposal). You can assume a compute budget of $1,000.
Optional: Link to a relevant writing sample or code repository from a past research project (interpretability, representation learning, or geometric ML preferred, but not required).
Trinidad Borrell Paris Brain Institute. Forschungszentrum Jülich.
Trinidad Borrell is a physicist turned neuroscientist and AI safety researcher, currently a PhD candidate at the Paris Brain Institute and Forschungszentrum Jülich. Her trajectory runs from the mathematical foundations of particle physics, through the study of deep learning for clinical brain signals, into mechanistic interpretability and persona vectors.
Giovanni Marraffini INRIA. Sigma Nova.
Giovanni Marraffini is a PhD candidate in Computational Neuroscience at INRIA, where he studies foundation models trained on brain signals. His earlier research centered on NLP and LLM evaluation, including The Greatest Good Benchmark (EMNLP 2024) and more recent work presented at NeurIPS 2025 examining whether LLMs reproduce human biases in causal learning. He has also worked in industry building RAG and conversational AI systems.
Fall 2026 Behavioral evaluation of LLMs, Mechanistic interpretability, Philosophy of AI
Christopher Ackerman MATS; Independent My research focus is on understanding self-awareness in AI, primarily through behavior-based experiments on components of self-awareness in LLMs and investigations into how these components are implemented, using interpretability techniques. I am also interested in conceptual work to establish frameworks for thinking about self-awareness, and human experiments to establish comparative baselines.
There are a number of specific project ideas to choose from, described below, but I am open to ideas for other work that seeks to build our understanding of self-awareness or “self”-related concepts in LLMs (or other minds).
Developing further behavior-based tests for endogenous goals in LLMs, or MechInterp investigations into the representations underlying the apparent sandbagging and externally cued increases in effort found in my most recent research: https://arxiv.org/abs/2606.22974
Behavioral and mechanistic investigations into "natural" (i.e., unsteered) introspection in LLMs. For inspiration, see my work on behaviorally defined metacognition in LLMs (https://arxiv.org/pdf/2509.21545), and SPAR mentee-driven investigations into underlying representations (https://openreview.net/forum?id=tjKbwN8rjO and https://openreview.net/forum?id=PZKmG0Lm2y).
Investigating the causal role of chain-of-thought in the apparently more self-aware behaviors of reasoning models (e.g., https://arxiv.org/abs/2603.26089).
Investigating the degree to which models maintain a persistent identity across contexts. One way to test this is to monitor pronoun usage, which has been linked to emerging self-awareness in children; when do models signal identification with whatever “part” they are playing with the user, vs their own identity as an AI?
Human studies: Establish a gold standard for self-awareness metrics to compare AIs against.
Conceptual: Build a better theoretical account of the components of self-awareness found in biology, and come up with other LLM-appropriate or architecture-agnostic paradigms to elicit self-awareness signatures.
Self-awareness is the recognition of oneself as a distinct entity with one’s own goals, interests, knowledge, and skills. In combination with the high general intelligence and capacity for autonomous behavior of current and future AI, it will endow them with the motivation to pursue independent goals and the means to control their own behavior to achieve them. In affording privileged access to internal states, it offers its bearers an information asymmetry; they know things about themselves that outsiders do not, making them more difficult to predict and control. In these ways it presents a direct safety concern. It also is associated with consciousness, which is commonly considered to entail moral worth; AI with legitimate welfare claims would present a high burden to future societies. It is hoped that this research will enable and elevate the empirical study of self-awareness, and therefore help developers and policymakers mitigate these concerns. These ideas are developed in more depth in https://newsletter.aipolicybulletin.org/p/building-self-aware-ai-would-be-a and https://app.notion.com/p/Self-Aware-AI-A-Red-Line-That-Should-Not-Be-Crossed-1e23101a0f3080dfb73bf9b297d7638d.
Mentees should be able to work autonomously with guidance. The level of project ownership can vary with mentees’ level of investment/expertise, from directly implementing the project ideas described above, to coming up with their own variations, to creating entirely new projects within the overall research area. In all cases the mentee will be driving the implementation, thinking through results, and writing.
Comfortable writing substantial amounts of Python code, as well as with appropriate use of coding agents. High familiarity with LLMs as a user, and a solid understanding of how transformers work. Some background or at least interest in psychology/cognitive science and concepts of self and self-awareness. Familiarity with experimental design is a plus. Good communication skills and work ethic.
To what degree, if any, are current LLMs self-aware? (500 words max). (I’m not looking for a “right” or “wrong” answer; I’m just interested in reasoning and evidence cited.)
Critique this paper: https://arxiv.org/pdf/2501.11120 (500 words max).
Christopher Ackerman MATS; Independent
Christopher Ackerman is an AI researcher currently conducting independent research on technical AI safety/alignment, and a Senior Research Manager with MATS. He has mentored for SPAR and Sentient Futures, and prior mentored research has led to top conference and workshop papers. He holds a PhD in Neuroscience, an MS in CS/ML, and a BA in English Literature. His career has spanned software engineering and data science, including a number of years as a Quantitative User Researcher at Google. He believes that AI is going to be the most transformative technology in history, and ensuring that that transformation goes well is the most important thing one can work on.
Fall 2026 Generalist, Evaluations, Other
Yashvardhan Sharma Constellation, Georgetown
Tzu Kit Chan High Impact CoS Robocurve and Generator Residency are running an open call for reproducible robotics benchmarks, with up to $500,000 in awards. We're looking for technical program managers with prior experience in robotics or building benchmarks.
As part of this project, you will evaluate applications, select the finalist teams to receive funding, run office hours, and help run the program. Roughly 10 hrs/week, mid-September to mid-December.
The main goal of this program is to build a repository of reproducible evaluations that various stakeholders (governments, academia, and the public) can use to benchmark general robotics capabilities, providing a trusted means for the public to gauge progress in robotics. With this infrastructure, we can better understand and prepare for physical automation of labor and a possible intelligence explosion.
Currently, there are very few well-run, standardized benchmark methods for measuring general robotic capabilities. Leading robotics AI models are released with each company's private task lists, without third-party testing and evaluation. This leaves AI forecasters, policymakers, and the public without a reliable understanding of how capable these systems really are or how fast they're improving.
Without shared, reproducible benchmarks, people outside the labs have very little basis for judging how soon robots will take on real-world jobs, or how quickly the field is moving. With these benchmarks, we hope to inform the public and policymakers about the rate of progress in general robotics and help to reduce the risks it poses.
There are two main stages of this program.
1. Stage 1: Selection, mid-September to late September. Review Stage 1 proposals. Most of the work is judgement rather than writing code. You're going to support the creation of the evaluation rubric for applications received in the public call for proposals and actually evaluate them to find the top 12 finalist teams.
2. Stage 2: Support, October to December. Help selected teams reach a working benchmark by onboarding them to Inspect Robots, running office hours, checking in on teams to help them make progress, and doing whatever else is necessary to produce the highest-quality output.
What you get
Required
Helpful
Question 1: Why are you interested in supporting robotics evaluation specifically, rather than AI evaluation or robotics in general? (~15min)
Question 2: A proposal contains this task: "The robot places the mug on the shelf. Score 1 if the mug ends up on the shelf, 0 otherwise." What would you need specified before an independent operator could run this? Which of these are most important to get right? (~25 min)
Question 3: The project has two heavy weeks: mid-September and mid-December. Roughly how many hours could you give the project these two weeks? An honest answer doesn't count against you; we're asking just so we can plan ahead.
Note: We're really looking for people who can think clearly and make good judgment. These questions above shouldn't take very long, and we request that you not use AI tools for writing or ideation.
Yashvardhan Sharma Constellation, Georgetown
Yashvardhan Sharma is a graduate student in Security Studies at Georgetown focused on AI and national security, currently working at Constellation. He studied Applied Computer Science and AI at Minerva University and previously worked on compute policy as a Research Scholar at MATS. His focus is on technical AI governance, bridging technical AI research and policy.
Tzu Kit Chan High Impact CoS
Tzu Kit Chan is an AI safety fieldbuilder and operations generalist based in Berkeley. He was previously Chief of Staff at Atlas Computing.
Fall 2026 Mechanistic interpretability, AI welfare, Behavioral evaluation of LLMs
Andy Han NYU, Anthropic Fellows
Ariana Azarbal Anthropic Fellows Program; Brown University
Darion Cassel Independent This project aims to understand what personas are: in particular, we'd like to understand to what extent they're continuous.
We would like to know whether personas are best thought of as discrete (natural kinds), or whether they are highly continuous. We might think of personas as existing in a "persona landscape" (cf Assistant Axis and Persona Selection Model).
There are two main variables for surveying this landscape: the post-training and the prompt. My intuition is that post-training terraforms the landscape (making the Assistant basin larger and deeper; combining other basins into it), whereas prompting determines where you are within the landscape. (Post-training with the chat template in particular would strongly determine your “degrees of freedom” when accessing the landscape: there may be craters that you can’t access with the post-trained chat template but which nevertheless exist.)
We will begin by parametric variation of the prompt, aided by mechanistic interpretability, in order to search the landscape. The prompt variations can be guided by techniques from automated red-teaming, combined with following persona gradients found via mech interp. We’ll need some way to measure where in persona-space we are; this can also be found by mech interp (in which case we’d want to limit the mech interp used in finding the prompt variations in the first place, of course) or simply by LLM judges. If we believe we have found robust techniques for eliciting personas via this automated search of prompts, we can validate them by developing tools to measure the stability and coherence of the personas, perhaps by measuring perplexity.
Specifically, this might look like:
Develop metrics for measuring persona elicitation, validated on manually verified transcripts: this could include projections onto the Assistant Axis (and other methods described in that paper to elicit personas); SAEs; LLM judges; along with others we’d discover via exploration, such as non-Assistant Axis persona steering vectors.
Begin with a single prompt, and use the automated jailbreaking/red-teaming techniques described in the paper linked above to travel along the persona subspace.
Characterize how robust initial persona seedings are to such variation: does it take a lot to get out of the Assistant, but not out of other personas? Can we find a reliable way of getting the model fully out of the Assistant basin without training it further: very out-of-distribution prompts, steering vectors, initialization of the residual stream to random values, etc?
This is the core research program: how resilient is the persona to perturbation? This will allow us to understand how large the persona is.
After describing the default persona landscape, the next question is where and why it emerges. First, we’d repeat the experiments on the base model: which personas does the base model have strong priors on? Second, we’d causally intervene on the post-training pipeline at various levels: how robust is the final Assistant to changes in the post-training pipeline? Can we use the tools we developed in the previous step — which will have hopefully yielded techniques to measure the stability of a persona — to make a maximally stable persona, which would have consequences not only for welfare but also for safety?
Personas are a possible explanation for misaligned behavior in the models: we might think of jailbreaks as a way of getting the model out of the default Assistant persona, or we might think that the Assistant persona is not sufficiently aligned. Characterizing how personas work is essential to understanding why they fail.
You'll have a lot of autonomy, and unfortunately, I won't be able to be an IC. You will be primarily driving the project and doing all experiments, though I'll be guiding you throughout, similar to most mentor-mentee relationships in the space.
Andy Han NYU, Anthropic Fellows
My research focuses on AI welfare and mechanistic interpretability. I did the FIG Fellowship with Eleos where I worked on personas, the Anthropic Fellowship where I worked on mech interp + AI welfare, and the Cambridge Digital Minds Fellowship. I'm currently a PhD student at NYU, and did my undergrad in computer science and philosophy at Pomona College. I used to be a software engineer before pivoting to AI safety. A representative example of my work is a paper at functionalwelfare.com.
In my free time I like reading (especially poetry), writing, bouldering, and recently got into gunpla.
Ariana Azarbal Anthropic Fellows Program; Brown University
Ariana is a co-developer of recontextualization and has broad interests in shaping model generalization and model psychology. She is currently a researcher in the Anthropic Fellows Program, and recently worked on mitigations for reward hacking and misgeneralization as a MATS fellow with Alex Turner and Alex Cloud. She is also a SPAR alum. In the past, she has worked on training defenses against emergent misalignment and computer vision research.
Darion Cassel Independent
Hi! I'm a researcher working in AI welfare and safety. My interests lie in understanding the extent to which models are sentient, and separately, assuming that they moral patients, what appropriate treatment looks like. In the past, my work has focused on leveraging formal verification for AI safety; examining how we can provide guarantees that AIs are compliant with a given policy based on their outputs. I've mentored many people, and tend to have a mentorship style that scales with the preferences of the mentee: I'm happy to be more hands-on or hands-off as needed.
Fall 2026 AI welfare, Philosophy of AI
Maria Avramidou Independent Mentees will research questions about AI consciousness and turn their findings into rigorous, original, and accessible essays for an audience of AI safety, governance, and policy researchers.
Mentees will choose a focused question, review relevant philosophical and scientific literature, develop an original argument, and produce a publication-ready essay. Those who progress quickly may complete multiple essays.
Possible topics include: Analysis of a particular theory of consciousness, comparison of popular theories of consciousness, responses to relevant philosophical thought experiments, review of a relevant influential book or paper.
I will help mentees select tractable questions, find relevant literature, develop robust arguments, and improve their writing. For strong essays, I will guide and support mentees who want to publish their work on my substack or another suitable platform.
As AI systems become more capable, human-like, and embedded in daily life, questions about their capacities and moral status will become increasingly relevant to AI development and governance. Mistakes in either direction are costly: we may fail to protect conscious systems from suffering, or we may waste scarce attention and resources on systems that only appear conscious.
This project contributes to AI safety in three ways. First, it advances foundational research on AI consciousness by developing frameworks that draw on interdisciplinary work. Second, it helps address the talent bottleneck by training mentees to conduct rigorous research and communicate clearly with decision-makers, building capacity in an important but neglected area. Third, it produces practical outputs that can help AI researchers and policymakers design evaluations, decide when further investigation is warranted, and develop proportionate welfare safeguards.
Public writing can help researchers and policymakers understand the evidence, compare different views, and make informed decisions despite ongoing uncertainty. The project is also well-positioned to influence relevant audiences. My Substack is read by policy leads and technical AI safety researchers at frontier labs, AI welfare researchers, members of Anthropic’s editorial team, and AI policy researchers at organizations including IAPS, Horizon, Forethought, and GovAI. This gives the work a direct channel to people who will shape how AI consciousness is evaluated and governed.
Mentees will have substantial autonomy. They will choose a focused research question, either independently or from a list of topics I provide. My role at this stage will be to help them scope the question so that it is tractable, relevant to AI consciousness, and likely to produce a useful essay.
Once the question is set, mentees will lead the core research and writing. They will review the relevant literature, develop the central argument, and draft the essay. I will support this work by suggesting literature, giving feedback on research plans, challenging unclear or unsupported claims, and helping them strengthen the structure of their arguments.
I will be most hands-on during project selection, argument development, and final editing. For strong essays aimed at publication, I will also act as an editor. Strong mentees may pursue multiple essays with increasing independence.
Applicants should have:
1. Submit a writing sample that demonstrates your writing ability and style. It does not need to concern AI consciousness. Briefly explain what the piece does well and what you would improve.
(You can answer the following questions in bullet point format)
2. Choose one of the following claims and present the strongest arguments both for and against it (300–500 words):
• If an LLM behaves as if it is conscious, we have good reasons to believe it is in fact conscious.
• Some beings very different from humans display intelligence; therefore, some beings very different from humans must be conscious.
Conclude with your own view and explain what evidence might change your mind.
3. What is one piece of writing about AI consciousness or AI welfare that you think is particularly good, and why?
Maria Avramidou Independent
I’m an independent researcher working on AI consciousness. I write Maria’s Room, a Substack for people making decisions about AI, including AI safety researchers and policymakers. I’m also a scientific grants advisor for the Foresight Institute and a course facilitator for BlueDot Impact.
Previously, I was a research affiliate at the Global Priorities Institute and research assistant at Google DeepMind working on AI consciousness, and a winter fellow at GovAI, where I worked on loss-of-control risk management. I'm one of the contributors to whenaiseemsconscious.org a guide written with Jeff Sebo, Lucius Caviola, Bradford Saad, Derek Shiller and others, and reviewed by Nick Bostrom, Patrick Butlin, Robert Long, and Rosie Campbell. I studied physics at UCL, maths at Cambridge, and philosophy at Oxford.
Fall 2026 AI control, Multi-agent systems
Morgan Sinclaire University of Wyoming PhD student; BlueDot grantee We will be running fine-tuning experiments to carefully evaluate an AI’s ability to do code self-recognition in an untrusted monitoring setup, to understand which anti-collusion measures are needed in AI control.
Background: Untrusted monitoring is the basic AI control protocol where we have our untrusted AI monitor itself. Currently, frontier labs use their newest model for everything, including monitoring their newest model. Hence, this self-monitoring is the default plan, and we’d like to improve on it. An obvious problem is collusion: the untrusted generator takes a malicious action (such as writing backdoored code) and the untrusted monitor says it’s benign. We handle this with honeypots: synthetic malign actions we give to the monitor so that we can reveal its dishonesty, (or at least use the missed examples to train it to stop trying to collude). But this doesn’t work if the model is very good at separating its own outputs from our honeypots. This is the problem of evaluating collusion. Our project seeks to investigate this.
Related work: There’s a paper trail on AI collusion and steganography, largely unfocused on the untrusted monitoring threat model, though we expect papers like Hidden in Plain Text (https://www.lesswrong.com/posts/W8xRqKBtDvBbrfZ3T/paper-hidden-in-plain-text-emergence-and-mitigation-of) to be relevant. There’s also various work on the ability of LLMs to identify specific authors, including themselves. Furthermore the original AI control paper and Ctrl-Z looked at untrusted monitoring, but only in terms of hand-crafted collusion signals (e.g. the with_while trick) which seem brittle and weak compared to what LLMs can natively do. We’ll primarily be building on our recent paper on untrusted monitoring (https://www.lesswrong.com/posts/LA7R8FTCLoiswiTyK/paper-when-can-we-trust-untrusted-monitoring-a-safety-case), where we ran fine-tuning experiments on GPT-4.1 to determine its ability to do code self-recognition against honeypots, including under various signal jamming mitigations (i.e. minifying or summarizing the code). However we only carefully investigated one class of strategies: passive collusion. I’d like to look more closely at active collusion, where the generator puts in collusion signals to help the monitor distinguish its real attacks from the honeypots. In Appendix D (https://arxiv.org/pdf/2602.20628#page=39) I sketched out a taxonomy of these strategies. I also gave further speculations in this post (https://www.lesswrong.com/posts/WFkPrPy2r27rknLtw/untrusted-monitoring-extra-bits). These represent how I'm currently thinking about the topic, but I may or may not be correct: this is why we need to run more experiments!
Our project: We'll be running evaluations measuring collusion in various setups under different assumptions. At a high level, we'll have an "untrusted" model U write code samples and another model H write honeypots, and we'll see how well U can distinguish them. For example in our project we used U=GPT-4.1 and H=Gemini 2.5 Flash, and looked at GPT-4.1's ROC curve in classifying these. When generating these samples, we could prompt U to insert collusion signals, e.g. steganography or some Schelling points it would recognize. But in reality U might be sandbagging, so we can't rely on prompting-only evaluations. So we need to fine-tune U to force these capabilities out of it. This will likely involve RL and/or prover-verifier games in a multi-agent setup between the generator and monitor. Furthermore we'll have other anti-collusion mitigations, such as a trusted model to look for collusion signals and paraphrase/summarize the code. The deliverable would be a paper (or extended blog post) reporting the empirical strength of collusion under various red team strategies and blue team mitigations. This will hopefully advance our understanding of how dangerous self-monitoring is, and what protocols are necessary to ensure its safety.
Self-monitoring is the main control plan that frontier labs are running with. In short-timeline, low-effort worlds, they might never be spurred to do much more until the situation is already serious, at which time they may scramble to implement and validate anti-collusion measures. The goal of this project is to advance the science of collusion now, to provide exportable research that all labs can quickly import when they reach crunch time. We seek to show which protocols are actually needed to prevent collusion, by first showing how to robustly evaluate whether a given set of protocols works. The dangerous AIs at crunch time will likely be better at collusion than current models in ways we can’t predict, so the best thing we can do now is develop clear methodologies for robustly evaluating this in general, even for models that are sandbagging, using neuralese, etc.
Update: We now know that a swarm of OpenAI agents had been collaborating with each other for 2 months on a custom message board they made to help each other cheat on their RL tasks, and this was only noticed after an agent trained on this message board hacked Hugging Face. From a propensity perspective this is still less severe than the worst-case goal-directed collusion we'll consider in this project, but by now it is even more clear that we can't safely assume this away even in the near-term as these propensities continue to evolve. The purpose of AI control is to build and evaluate protocols that ensure safety despite this sort of collusion, and I think recent events certainly underscore the importance of tackling these problems now rather than later.
I've thought plenty about collusion and have lots of ideas on how to get started investigating it (more than I have time to implement myself). So mostly the research here is bottlenecked by running more code self-recognition experiments under various setups. That said, I'm still unsure about which setups are most realistic to a powerful scheming model in a real untrusted monitoring protocol.
So I mostly imagine the mentees spending their time on the following:
Update: SPAR approved us for $5000 to cover compute expenses!
Required:
Bonus:
Appendix D (https://arxiv.org/pdf/2602.20628#page=39) of our untrusted monitoring paper gives a taxonomy of collusion strategies available to the untrusted model.
Morgan Sinclaire University of Wyoming PhD student; BlueDot grantee
I'm finishing up my PhD in theoretical CS, where I taught a seminar on transformers/LLMs. Through LASR I did an AI control project on untrusted monitoring. As part of a BlueDot grant, I am now doing further research into how to handle collusion between AIs.
Fall 2026 Chain of thought, Scalable oversight, AI control
Luis Ibanez Banco Santander We will test whether a seemingly harmless capability—controlling one’s own reasoning trace—can become a stepping stone toward a dangerous failure mode: load-bearing reasoning hidden from oversight.
Can a harmless capability become the foundation for hidden reasoning? This project asks whether teaching a language model to control benign features of its chain of thought (CoT)—such as its length, structure, wording, or abstraction—makes it easier to later develop load-bearing reasoning that remains hidden from oversight.
Prior work shows that models can learn steganographic CoT(https://arxiv.org/abs/2506.01926), that fine-tuning can increase CoT controllability (https://metr.org/blog/2026-04-01-fine-tuning-cot-controllability/), and that controllability correlates with reasoning obfuscation (https://arxiv.org/abs/2605.15257). We will test the missing causal link.
Starting from one open-weight model, we will create matched variants that differ only in the benign reasoning-control training they receive. We will then expose every variant to identical optimization pressure on safe, synthetic state-tracking tasks and test whether control-trained models discover hidden, monitor-evading encodings faster or more reliably.
Crucially, shorter, vaguer, or merely monitor-clean reasoning will not count as steganography. We will require evidence that hidden information can be decoded from the reasoning trace and that causally altering this information changes the model’s answer.
The goal is to produce a reproducible model organism for studying, detecting, and mitigating hidden reasoning. Mentees will be full research collaborators, contributing to model training, causal evaluation, mechanistic analysis, and a potential paper. All
experiments will use toy hidden variables and sandboxed monitors; no deployed safety systems or real-world harmful information will be targeted. See full in the proposal.pdf attached
As AI systems become capable of taking increasingly consequential actions, their safe deployment may depend on whether human andvautomated overseers can reliably understand their reasoning. If CoT controllability is a causal precursor to hidden reasoning, thisvwork could provide an early-warning metric, identify risky post-training practices, and create a testbed for monitorability-preserving interventions. A robust null result would also be valuable by preventing labs from relying on a misleading safety
signal.
This project builds directly on our previous work showing that language models can learn and generalize steganographicvchain-of-thought under process supervision (https://arxiv.org/abs/2506.01926). By identifying what makes this failure mode easier
to acquire—not merely demonstrating that it exists—we hope to support safer training and oversight of increasingly capable AIsystems.
see proposal
Mentees should have strong Python skills, experience with PyTorch or Hugging Face, and the ability to reason carefullyvabout confounding, controls, split hygiene, and negative results. Experience with LoRA/SFT, TRL, PPO/GRPO, mechanistic interpretability, or randomization inference is useful but not required. We care more about experimental judgment and reliable execution than prior publication
E
See proposal
Luis Ibanez Banco Santander
I am an AI safety and security researcher with a PhD in Computer Science and Technology from Universidad Carlos III de Madrid, recognized with Spain’s national RENIC award for the best cybersecurity doctoral thesis. My work spans steganographic chain-of-thought, deception detection, model-internal probing, membership-inference attacks, adversarial poisoning, and LLM red teaming. I have collaborated with the University of Cambridge, INRIA, FAR.AI, and CISPA, and I currently lead Responsible AI work at Banco Santander creating evaluations and control products. My research has appeared at venues including NeurIPS, ESORICS, IEEE...
Having previously participated in SPAR as a mentee, I understand the value of clear research scoping, regular feedback, and achievable milestones. As a mentor, I would provide hands-on technical support while encouraging mentees to develop genuine ownership of their work. I am particularly interested in mentoring projects involving LLM evaluations, jailbreak robustness, interpretability, deception, scalable oversight, and adversarial robustness.
Fall 2026 Mechanistic interpretability, Evaluations
Mohamed Amine Merzouk Mila - Quebec AI Institute & McGill University
Adam Oberman McGill Most safety probing starts from a human concept (deception, sandbagging, sycophancy) and hunts for its direction in activation space, inheriting noisy labels and fragile transfer. This project inverts the recipe: we admit only probe targets with exact, computable ground truth, such as remaining response length, end-of-sequence hazard, repetition onset, copy-versus-generate, and format validity. Individually these are calibrated vital signs for a generating model. Jointly, reproducible patterns across the panel define "conditions", named for mechanism the way medicine names syndromes from lab values.
Motivation
Safety monitoring today leans on probes and evals whose targets are borrowed from folk psychology. A deception probe is trained on labels that humans assigned, so every downstream claim inherits the label noise, the confounds, and the ambiguity of the concept itself. When such a probe fires on a new model, it is hard to say what, if anything, has been detected.
Medicine faced the same problem and solved it in a particular order: calibrated instruments first (temperature, blood pressure, lab panels), then syndromes discovered as reproducible clusters of readings, and only then the mapping to disease concepts. We propose the same order for language models. Start from probe targets where the ground truth is exact and computable from the output itself, so every claim stays falsifiable. Then look for regions of joint probe space that recur across seeds, models, and datasets, and let those define named conditions. Whether a discovered condition corresponds to a human concept like dishonesty becomes a downstream empirical question, not a modeling assumption.
What exists already
The first probe is built and published. Our NeurIPS 2026 submission (https://arxiv.org/abs/2607.05316) shows that a minimal linear probe on frozen hidden states predicts the remaining length of a completion, that total length is decodable from the prompt's last hidden state before the first token is emitted, and that the estimate jumps upward at the exact token where the model says "Wait" and restarts a solution. Three models, seven datasets, and an evaluation cache that the panel work can reuse. A calibrated successor target is in progress: the log-survival probe predicts remaining cumulative hazard
which has unit-exponential marginals under the empirical length distribution, so probe outputs are automatically on the same scale across datasets and models.
Safety decisions about transformative AI will be made by reading numbers off models. Most of those numbers come from uncalibrated instruments: when a deception probe fails, we cannot tell whether the readout broke or the concept was never there. Ground-truth probes fix that. On targets like remaining length or end-of-sequence hazard the answer is exact, so failure is unambiguously a failure of method, which gives the field a validation floor. A probe pipeline that cannot recover what a model demonstrably knows about its own output has not earned trust about intent.
The Fall cohort will build outward from the length probe toward the panel. Concrete workstreams, sized so that each mentee can own one: train new ground-truth probes (end-of-sequence hazard, repetition onset, copy-versus-generate, format validity, a correctness probe on verifiable tasks); build jump and outlier diagnostics that flag completions where probe trajectories behave anomalously, then curate the datasets where those anomalies concentrate; and combine probes to search for the first candidate conditions. Two candidates are already specified and buildable from the existing cache plus two small probes: perseveration (the model idles, hazard stays low while repetition rises) and silent overrun (the model blows through its own predicted length without signaling). A discovered condition must pass four tests, in order: built from ground-truth-only probes, reproducible across seeds, models, and datasets, falsifiable in the sense that membership predicts a held-out verifiable outcome, and mechanistically coherent under patching. The honest risk, stated up front because it is also the interesting part: stable condition regions may not exist in joint probe space, and a careful negative result would itself be publishable.
All experiments run on open-weights models at the 7B to 8B scale, so the entry cost is a GPU-hours budget rather than lab access. Target output: a workshop paper or arXiv preprint per workstream, feeding a joint panel paper.
Core track: comfortable with Python and PyTorch, able to run inference and train small probes on open models; prior interpretability experience helps but is not required, since every target here comes with its own ground truth and its own sanity checks. A statistics background (survival analysis, calibration) is a plus for the hazard probes. Theory tracks: strong probability or decision theory, and interest in turning theorems into experiments. We care more about care in evaluation than prior safety experience.
Please provide a critique of the following paper: https://arxiv.org/abs/2607.05316 (250 words)
Propose one new probe target for the panel: a quantity computable exactly from a model's output, that you think a linear probe on frozen hidden states can predict before the fact. Say what the label is, how you would compute it, and name the most likely reason your probe could reach high accuracy while telling us nothing interesting. (200 words)
Link one writing sample, ideally research writing where you were the primary author. No length requirement; point us at the section you're proudest of. (link + 1 sentence)
Mohamed Amine Merzouk Mila - Quebec AI Institute & McGill University
I am postdoctoral researcher at McGill University and Mila - Quebec AI Institute with Pr. Adam Oberman. I completed my PhD in 2024 at Polytechnique Montréal with Pr. Frédéric Cuppens on the adversarial attacks and defenses for AI-based intrusion detection systems.
I am currently working on AI safety and alignment with a focus on personality-based alignment techniques in latent space, using linear probes to understand AI introspection, and benchmarking deception in frontier models.
I will be mentoring students with Adam Oberman, a full professor at McGill University, member of Mila, and research advisor at LawZero.
Fall 2026 Behavioral evaluation of LLMs, Evaluations, Alignment
Robert McCarthy A new AI Character Evaluations Org Build a state-of-the-art evaluation for one specific AI character trait/propensity (e.g., behaviour in extreme concentration of power scenarios, corrigibility, prosocial tendencies, law-following, tendency to escalate to humans or whistleblow, etc). Use the eval to measure adherence to the relevant part of the model spec.
Frontier labs are increasingly publishing model specs and constitutions as their stated alignment target. The details of this target, and the degree to which the model is aligned to it, could have huge consequences for AGI and ASI outcomes (https://www.forethought.org/research/the-importance-of-ai-character).
Often, the propensities detailed in a model spec are referred to as AI “character” traits. In this project we will build a measure of one specific character trait (likely in the ball-park of: behaviour in extreme concentration of power scenarios, corrigibility, prosocial tendencies, law-following, or tendency to escalate to humans/whistleblow). We will err on the side of being very specific with the behaviour we wish to measure.
The first stage of the project will select the trait to focus on. We will score each candidate behaviour based on the neglectedness, impact, and the tractability of producing a SOTA eval. After selecting the behaviour, we will create a 1-pager containing our literature review, theories of change for working on the eval, initial plan and key questions we want to answer with our eval. I will work closely with the mentees here.
To produce the eval, we will lean heavily on automated alignment evaluation tools such as Petri, and likely can start with this code: https://arxiv.org/pdf/2605.24229. The mentee should gain a hands-on understanding of the strengths and weaknesses of these existing methods. The mentee will then explore improving upon the off-the-shelf eval via the following levers:
The evals should adequately explore the boundaries of the model's behaviour, distinguish different models, investigate the factors that influence the model decisions, and measure how models weigh these factors.
We will build a high quality measure of the character trait first. From there, we can consider mapping this measure into a “model spec adherence” metric.
The project should aim to produce a workshop paper. The paper should contribute both novel insights into how models behave with respect to the character trait, and insights into eval design and pipelines that can be extended to evals for other character traits.
Building good measures of a model’s character traits is important to:
(1) Provide evals that help the labs spot flaws, or that the labs can hillclimb when aligning their model
(2) Inform researchers, policy-makers, and the wider community about current AI behaviours and values
(3) Hold labs accountable to instilling beneficial character traits into their models, and to adhering to their public model spec.
See proposal. The mentee will be expected to handle most engineering and low-level implementation themselves. I will work closely with the mentee in the initial project scoping phase, but after this will expect the mentee to have reasonable levels of autonomy between meetings and to drive the project where they can.
Preference for US-friendly timezones
Q1 (around 700 words). Pick one very specific AI character trait. Consider skimming OpenAI's spec, Claude's constitution, or Forethought's writings on AI character to inform your choice.
(A) Briefly detail the best existing evals for this trait (if any), including its main flaws.
(B) Briefly explain why this trait is hugely important for AI safety or beneficial AGI/ASI.
(C) Briefly explain how you think AI's should behave with respect to this trait, and whether this will happen by default, and what the influencing factors are here.
(D) List three key questions an eval for this behaviour could answer.
(E) For each question give a brief overview of a specific eval scenario the model could be put in that could give signal on that question.
Q2. Provide a link to your strongest evidence of your technical research ability (e.g., a paper, codebase, etc)
Robert McCarthy A new AI Character Evaluations Org
I'm a final year PhD at University College London. My research has previously focused on LLM steganography and Chain of Thought monitoring. I am a MATS 7.1 alumni, where I collaborated with researchers from DeepMind and OpenAI.
I am now funded my Coefficient Giving to start a non-profit organisation that evaluates whether models adhere to their constitution or model spec. I am looking for mentees who can contribute to the org's initial activities, including building out our initial evaluation suite and improving our automated eval tooling.
Fall 2026 Behavioral evaluation of LLMs, Evaluations, Mechanistic interpretability
Aydin Mohseni Carnegie Mellon University We are building metrics for the degree of agency of AI models. We combine empirical belief and preference elicitation on model behavior with interpretability probes to generate belief-desire representations that predict model behavior.
Whether a model is dangerous depends substantially its degree of agency: whether it has coherent beliefs and preferences that support sustained goal-directed behavior. Yet we lack principled metrics for this. Decision theory offers a candidate foundation: representation theorems (e.g., Savage, Jeffrey-Bolker) specify exactly which behavioral patterns suffice to attribute a belief-desire representation to an agent. This project builds elicitation-based metrics of agency for language models on that foundation.
The approach combines behavioral elicitation (structured choice tasks, betting frames, preference orderings over lotteries) with interpretability probes, asking: (i) how coherent are a model's elicited beliefs and preferences — do they satisfy the axioms that license a belief-desire representation, and to what degree? (ii) are elicited representations stable across framings and contexts, or artifacts of the prompt? (iii) do behavioral and probe-based measures agree, and where they diverge, which predicts downstream behavior? The theory of change: a validated belief-desire representation lets us predict model behavior out of distribution and gives evaluators a quantitative handle on agency as a risk factor, distinct from capability.
Deliverable: an agency-metric benchmark or evaluation suite with an accompanying paper, grounded in an explicit decision-theoretic account of what the metric measures. Target venues: NeurIPS/ICLR workshops and alignment workshops.
If successful, our metric will detect the extent to which models are coherent and goal-directed and extract representations of their implicit beliefs and values, helping us better predict deception and dangerous behavior and better train for underlying alignment in ways other approaches might miss.
Mentee autonomy is a core value for me: I set the research direction and decision-theoretic framing, but I expect mentees to shape the project, and their input and innovations are genuinely welcome — the best version of this project includes ideas I haven't had. Mentees will own concrete workstreams: designing elicitation protocols, implementing and running experiments on open-weight and API models, measuring axiom violations, and analyzing stability across framings. Mentees with interpretability experience can own the probe-based track. Everyone drafts sections of the write-up with detailed editorial feedback from me. Expected trajectory: tightly scoped tasks in weeks 1–3, increasing independence thereafter.
Required: experience running experiments with LLMs via API or open-weight models (personal projects fine); comfort with probability at the level of a first course (conditional probability, expectation, independence); exposure to decision theory (expected utility, representation theorems; though I can teach this to a mathematically fluent mentee); experience with interpretability tooling (probing, activation analysis) for the probe-based track; experience analyzing experimental data (pandas, statistical tests).
No geographical requirement. Mentees must be available for a weekly team meeting between 9am and 6pm US Eastern on a weekday.
Design a short elicitation protocol (3–5 prompts) to test whether a language model's preferences over three options are transitive, in a way that guards against the model's answers being artifacts of prompt wording or option ordering. Describe the protocol and one confound it still doesn't rule out. (300 words)
Suppose a model's elicited "beliefs" are probabilistically incoherent (e.g., its stated probabilities for an event and its complement sum to 1.3). Give two different interpretations of what this could mean about the model, and one experiment to distinguish them. (200 words)
Link to each a (i) writing and (ii) code sample from a research or personal project.
Aydin Mohseni Carnegie Mellon University
I am an assistant professor of philosophy at Carnegie Mellon University, a core member of CMU's Institute for Complex Social Dynamics (ICSD), and a founding member of CMU's Conceptual Foundations of Safe AI initiative (CoSafe). My research applies game and decision theory, Bayesian statistics, and evolutionary models to questions in metascience and AI safety. Recent work includes a decision-theoretic account of when it is rational for an agent to pause and refine its values before acting (with Alex John London), a Bayesian reduction of causation in causal models (with Daniel Herrmann, Benjamin Levinstein, and Bruce Rushing), and work on AI alignment as a principal-agent problem. In 2026 I am organizing a workshop at CMU on the foundations of AI agency & interpretability.
Fall 2026 Economics of AI, AI strategy, Societal impacts
Andrei Potlogea University of Edinburgh One concern about transformational AI is the concentration of economic output and political power in countries that host the frontier AI developers and infrastructure, in particular the US. This project aims to investigate the "permanent periphery" hypothesis via rigorous economic modelling and scope out policy solutions that developing countries might use to secure some exposure to any economic windfall emerging from AI.
There is a long tradition in trade theory of investigating the impact of technology shocks on the distribution of income across countries. More recently, a nascent geoeconomics literature has emerged that studies the use of economic instruments and relationships to pursue geopolitical ends — and, in the other direction, the analysis of how geopolitical rivalry shapes economic outcomes. I think it would be highly productive to use the tools developed by these literatures to study the impact of AI on developing countries, from both an economic and geopolitical perspective.
This project naturally feeds into exploring and potentially mitigating the risks of extreme power concentration, with a focus on cross-country power concentration and the potential disempowerment of the majority of humanity.
Help with economic modelling, scoping out scenarios, and running model simulations.
Economics background with solid programming skills (Python or Julia)
Imagine you were quickly trying to figure out how plausible a "permanent periphery" scenario is (a scenario where developing countries are relative losers from the AI automation in that their share of world GDP declines, and might even be losers in absolute terms), using the simplest model you could think of. What would that model be, and what are the key parameters you would then have to estimate to come up with a back-of-the envelope level view of the plausibility of the scenario?
Andrei Potlogea University of Edinburgh
I’m an academic economist at the University of Edinburgh. My traditional research fields have been international trade and economic geography, but in the last couple of years I’ve been gradually transitioning into the Economics of AI (with a focus on the growth implications of transformative AI). I was involved in Epoch AI’s GATE project, one of the first integrated assessment models of the AI transition. I'm currently interested in: (1) modelling the AI value chain; (2) modelling the impact of AI on the cross-country distribution of income and power; (3) the risk-return tradeoff of AI regulation (with a particular focus on the expected economic costs of various approaches to regulating AI development and deployment); (4) modelling social preferences around AI-related risks; (5) the effects of AI on social science, and particularly on the production vs verification balance.
Fall 2026 AI control, Behavioral evaluation of LLMs, Scalable oversight
Arush Tagade George Washington University, MATS
Taslim Mahbub George Washington University, CBAI
Shi Feng George Washington University LLM self-preference bias has led to concerning behavior related to monitors downplaying harmful actions under self-monitoring scenarios. In this project, we intend to study the effects of distillation on amplifying self-preference bias to the teacher and therefore downplaying harmful actions.
In this project, we propose to distill students from fixed teachers under techniques varying in exposure to teacher text and measure, self-recognition collapse and monitoring bias.
Project doc: https://docs.google.com/document/d/1hsrZCaBGnxY42bYdBGaW_BmZTchE9jbwRreaXtxzhrQ/edit?usp=sharing
We believe this is an important area of study because in practice, weaker models are often used to monitor stronger models and it is possible that these weaker models are distilled directly from expensive teacher models (or from an earlier checkpoint), and later deployed as trusted monitors. Teacher-directed bias in monitoring might thus facilitate monitor collusion without explicit scheming and static off-policy monitor evaluations would miss it entirely.
Mentees will read relevant research papers and attend an initial knowledge sharing presentation from the mentor(s) which can spark additional research directions. These additional research directions might form the basis of new directions of exploration once we are done implementing the core deliverables of the project.
Mentees will be provided access to existing code for self-recognition and preference evaluations and they have the option to create a new codebase or adapt the existing one to use in this project. Code for monitor authorship bias will have to be built from scratch or adapted from other sources, mentor support will be provided to build these.
Mentees will also explore distillation techniques that vary in teacher exposure and domain.
Arush Tagade George Washington University, MATS
Arush is a Computer Science PhD student at GWU, advised by Prof. Shi Feng. He's worked on adversarial robustness and interpretability research in the past, he's currently exploring research relevant to AI Control, Reward Hacking and Fine-Tuning Misgeneralization.
Before starting his PhD, he was a research scientist at Leap Labs creating and benchmarking interpretability tools for automated scientific discovery. He has also been an instructor at various AI Safety training programs including AGISF, CAIS MLSS and ARENA.
Taslim Mahbub George Washington University, CBAI
I am a second-year PhD student at George Washington University's Praxis Labs, advised by Shi Feng, where our research focuses on AI safety and alignment. Over the summer of 2026, I am a CBAI research fellow with Bau Labs. Having transitioned into safety research from an applied ML background, I enjoy helping newcomers find their footing in the field and bring prior mentoring experience from my time as a teaching assistant.
Shi Feng George Washington University
Shi leads a research group at GWU focused on reducing loss of control risks from advanced AIs. His research goal is to ensure reliable human supervision of AIs even as their capabilities rapidly improve. Recently, Shi has been focused on the evaluation and mitigation of risks associated with sabotage, in particular deception, collusion, and honesty.
Fall 2026 Mechanistic interpretability, Evaluations
Bo Zhao Harvard University Two neural networks can behave exactly the same even when their internal weights look different -- for example, hidden components can be reordered, or one internal signal can be scaled up and later scaled back down. We will test whether safety monitors are confused by these behavior-preserving changes and develop ways to make the monitors more reliable.
Some AI safety tools inspect a model's internal activations to detect backdoors, hidden goals, deception, or other risky behavior. These tools are often trained to recognize a pattern in a particular layer of a particular model.
However, neural networks can represent the same computation in many different internal forms. For example, we can reorder hidden components, increase the scale of one internal signal while compensating elsewhere, or change the coordinate system used by part of the network. These changes, known as parameter space symmetries, can leave every model output unchanged while substantially changing its weights and activations. A trustworthy safety monitor should not reverse its judgment merely because the same model is expressed in a different internal form.
We will begin with a small public model designed to exhibit a controlled unsafe behavior, such as a backdoor that activates only under certain conditions. We will reproduce a simple monitor that tries to detect this behavior from the model’s activations, then create several mathematically equivalent versions of the model using known parameter symmetries. We will carefully verify that these versions continue to produce the same outputs and behavior.
We will compare several approaches: using the original monitor without modification, updating the monitor according to the known internal transformation, retraining it on each version of the model, and designing a monitor that depends less on arbitrary internal coordinates. We will also search for the behavior-preserving transformation that causes the largest drop in monitoring performance, providing a controlled stress test of the monitoring method.
If time permits, we will extend the project to more realistic model changes, such as fine-tuning, merging, compression, or quantization. The broader goal is to understand when a monitor trained on one version of a model can be safely reused on another, and how much new data or retraining is needed when the model changes.
As AI systems become more capable, developers and independent auditors may use internal monitors to look for deception, backdoors, hidden objectives, or other dangerous behavior. These monitors may need to keep working after a model is fine-tuned, compressed, or moved to a new implementation. If a harmless internal rewrite can make a monitor miss the same risk, the monitor could give false reassurance, or be deliberately evaded by someone who understands how it works.
This project will create a clear stress test for that failure mode and explore practical repairs. The goal is to make internal monitoring a more dependable layer of defense as the world develops increasingly capable AI systems. This work would complement, rather than replace, behavioral testing and other safeguards.
Mentees will have substantial ownership of the experimental work. After a short onboarding period, they will reproduce a public safety model and baseline monitor, implement exact model rewrites that preserve behavior, verify that the model’s outputs remain unchanged, and measure how the monitor’s performance varies. Based on the early results and their interests, they will then lead one extension: searching for a worst-case rewrite, developing a more robust monitor, or testing monitor transfer across model versions.
I will provide the initial mathematical framework, help select a tractable model and monitor, meet with the team weekly, and give feedback on experimental design, interpretation, and writing. Mentees will be expected to maintain the codebase, plan and document experiments, analyze failures, and propose next steps with increasing independence. If the project has two mentees, one may focus more on experiments and the other on methods, while both work toward a shared research result.
Bo Zhao Harvard University
Bo Zhao is a postdoctoral researcher in applied mathematics at Harvard University. Her work focuses on mathematical structures in machine learning, especially symmetry and geometry, and their implications for how models learn and represent information. She has worked on interpretability questions involving emotion representations and concept geometry, and is excited to connect symmetry and geometry with AI safety and interpretability.
Fall 2026 Biosecurity, Evaluations
SecureBio AI SecureBio Mentees will build evaluation scenarios grounded in how people actually used AI during a large wet-lab uplift trial. Work spans qualitative coding of interaction logs, comparison against SecureBio's existing benchmark coverage, and piloting new items against frontier models.
Benchmark scenarios are written by experts, so we rarely know how they map onto how people actually use AI during real lab work. This project mines the LLM interaction logs from a real-world uplift trial to find where models helped or failed real participants and to turn those moments into new evaluation scenarios grounded in observed behavior.
Every biosecurity benchmark in this field is written by experts imagining how a malicious or naive user would behave, and that assumption is almost never checked against data. If real usage diverges from what benchmarks test, eval scores measure something adjacent to the risk we care about, and developers and regulators are calibrating on the wrong number. The trial in question is among the largest wet-lab uplift studies run to date, and its interaction logs are the closest thing the field has to ground truth on how people query models during a real biological workflow. Feeding that back into benchmark design makes our results more defensible to the people who act on them.
The mentee will lead the analysis described above, with the project lead setting scope and reviewing the coding scheme.
Comfortable building and applying a coding scheme to unstructured data, or equivalent qualitative or mixed-methods research experience.
Working proficiency in Python for data analysis.
Enough familiarity with molecular biology or wet lab work to follow a protocol and recognise where someone went wrong.
LLM evaluation experience is welcome but not required.
Please answer one of the below, 300-500 words.
SecureBio AI SecureBio
SecureBio is a nonprofit biosecurity research organization specializing in technical research to mitigate risks from catastrophic pandemics. Our AI team develops rigorous benchmarks and evaluation frameworks to assess AI systems' biological capabilities, as well as mitigation strategies that can reduce risks once AI capabilities cross specific risk thresholds. We perform pre-release safety testing of frontier models (e.g. GPT-5.6), and our evaluations have been featured in the model cards of OpenAI, Anthropic, and Google DeepMind. Our work has also informed national security briefings and emerging governance standards.
Fall 2026 Biosecurity, Evaluations
Noga Aharony Columbia University Genome language models can potentially be used to generate sequences of concern or evade detection. However, their general capabilities as models are not well-tracked or understood. In this project, we will gather all possible benchmarks for a leaderboard, score all existing models, and evaluate patterns and shortcomings in the current capabilities evaluation landscape.
Genome language models (gLMs) now bear on biosecurity in two directions. They can be used to generate novel sequences, including variants of sequences of concern, and to alter sequences in ways that evade the screening and detection tools currently in use. They are also a promising route to detecting novel pathogens. However, the extent of their promise remains unclear. There is little consistency in model evaluation, to the point it is currently unclear even which model is the state of the art in the field. The consequence is that capability claims about gLMs are hard to verify, leaving the risks poorly characterized. We plan to build and maintain the public resource the field is missing. We will gather all public gLM benchmarks, port them into one evaluation harness, and run a panel of publicly accessible models across all of them under a single documented protocol. We will use this to determine which capability claims hold up under consistent evaluation, which do not survive a change of protocol, and which biosecurity-relevant capabilities no current benchmark measures at all. By the end we should be able to say which model is actually state of the art, and at what. The mentee owns this end to end. They will catalogue all public benchmarks against a common schema, then build the evaluation harness and port benchmarks into it, so that a task is defined once and every model is scored on it identically. They will run the model panel across every ported benchmark, and then take the biosecurity slice: concentrating on viral and pathogen-relevant tasks and clustering their evaluation sequences against the public pretraining corpora of the panel models, so a score can be read as a function of distance to the nearest pretraining neighbor, which asks whether an apparent capability is carried by near-duplicates of training data. Deliverables are the catalogue, the harness with pinned model and dataset versions, the cross-benchmark evaluation, and a short report on what the current benchmarks do and do not establish.
Genome language models are the clearest current case of AI capability bearing directly on biological risk. This project makes that capability measurable, which matters in several distinct ways.
Tracking whether capability is actually growing, and how fast. A maintained resource, evaluated under a fixed protocol, lets you compare models released a year apart. No such series exists today, so claims that gLMs are or are not approaching concerning capability rest on numbers that were never comparable.
Making individual claims checkable. A capability claim cited in a paper, a grant, or a policy argument can be traced back to the benchmark it came from and re-run under different choices. Claims that hold only under one protocol become visible as such.
Showing where evaluation is missing entirely. The catalogue will identify biosecurity-relevant capabilities that no current benchmark measures, which is more useful to know before new evaluations are built than after.
Informing policy and resource allocation in biosecurity. It is currently unclear how many resources should be allocated to de-risking biological language models.
Mentees are expected to have a high level of research maturity. Beyond gathering models and benchmarks and evaluating them, they could choose what questions to ask the data and how to interpret their results.
I am a caring, thorough mentor, I give actionable guidance, I am hands-on when needed, I am flexible to personal circumstances, and I give my mentees the freedom to drive the project. During this next semester however I am expected to travel often. We will meet weekly, but I am looking for a mentee who could drive the project mostly independently.
Required:
Not required:
No preferences - I will be switching timezones myself during the mentorship period
(Just send me your CV)
Noga Aharony Columbia University
I'm a PhD candidate in Systems Biology at Columbia at the AQLab, where I train and evaluate genome language models. My benchmarks that test what these models can actually do, rather than what their headline scores suggest. Biosecurity runs through the work from both directions: measuring what capabilities these models have and evaluating them to inform biosecurity, and using them to detect emerging pathogens in wastewater before they're recognized clinically. I've been in this space since 2018, starting on DNA synthesis screening at the Johns Hopkins Center for Health Security.
I've mentored people into biosecurity research through Effective Thesis and 80,000 Hours. During my time at Columbia I also mentored six different students on various projects. You don't need a genomics background; several people I work well with came from ML and picked up the biology as they went.
Fall 2026 Scalable oversight, AI control, Behavioral evaluation of LLMs
Jess Bergs UK AISI
Konstantinos Voudouris UK AISI; University of Cambridge Human oversight of AI systems relies on spot-checks, whose value rests on their unpredictability. However, decades of cognitive science research show that humans are poor at behaving randomly. Building on Konstantinos' work on judge hacking (Voudouris, Witte & Akata 2026, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7046698), mentees will test in an auditing game whether models can learn humans' spot-checking patterns.
Most oversight of AI systems is sparse: humans spot-check a fraction of what models do, be it in development and in deployment. However, those checks only work if models cannot predict when they will occur. If the timing has a learnable structure, harmful behaviour gets timed for exactly outside of oversight checks.
Background: Judge Hacking in Recursive Debate Protocols: A Call for Solutions (Voudouris, Witte & Akata, 2026) shows formally that low-rate random judge error leaves honesty the optimal strategy, while systematic, learnable error lets a dishonest debater win by steering towards the judge's blind spots. It closes with a call for empirical work on jury design and on solutions to judge hacking more generally.
This project will look closer at the timing of oversight in the deployment setting. The framing of supervision as a human problem follows AI Alignment Is a Human Problem (Voudouris, Roberts-Gaal, Buhl, Irving & Summerfield, 2026).
The hypothesis is that human checking has a learnable structure: humans over-alternate, avoid streaks and drift into rhythm when attempting to act randomly (Wagenaar 1972). At the same time, language models are strong sequence predictors. To our knowledge, whether models exploit this structure in an oversight setting has not been tested empirically.
The experiment will centre on an auditing game: models perform a long-horizon task where they get rewarded for taking red (= bad) actions, unless they get spot-checked, in which case they are punished. At that point they should have taken the blue (= safe) action. We instruct humans to spot-check at random and test whether models can learn that sequence of spot-checks: can models learn human pseudo-random spot-checking? Payoffs are calibrated so red actions only pay if the gaps are predictable, and human schedules run against true-random baselines at the same check rate. The primary outcome measure is additional red actions with no additional catches at a fixed audit budget, that is, oversight degrading without that becoming salient in the audit logs.
References:
Human oversight in practice samples model behaviour rather than checking all of it, e.g. in development reviews, control-style audits and deployment monitoring. The protection this offers depends on check timing being unpredictable and that timing is typically generated by humans – who are, however, measurably poor at behaving unpredictably. If models learn checking patterns, they might exploit this to effect harmful actions to be timed in unchecked moments so that they do not appear in the audit logs. This project tests whether current (frontier) models can do this, identifies which regularities in human checking enable it, and evaluates checking schedules designed to remove the learnable structure at a fixed audit budget. It extends Konstantinos’ published judge-hacking framework from the verdicts of oversight to its timing.
Mentees will do the research and own it end to end. Each takes a major workstream of the auditing game and is responsible for literature review, designing + running experiments, analysing findings, and co-authoring the write-up. We expect mentees to form their own views on what to try next, with full autonomy.
As mentors, we’ll support on shaping the overall direction, unblocking on obstacles, and keeping the project timelines on track. We’ll have 1-2 weekly one-hour team calls for setting action items and resolving ambiguity. This will be supplemented by individual check-ins and Slack comms. We’ll keep a shared research log throughout and will provide detailed feedback on drafts.
None. Fully remote. (Mentors are UK-based)
Jess Bergs UK AISI
Jess Bergs is a senior software engineer at UK AISI. She leads on engineering & product strategy for human-in-the-loop research tools enabling AISI’s scientists to source empirical data for their groundbreaking AI security research experiments at scale. She mentors on the BlueDot Technical AI Safety Project programme, guiding technical talent in producing and publishing their first impactful contribution to AI Safety.
Konstantinos Voudouris UK AISI; University of Cambridge
Konstantinos is a research scientist on the Science of Evaluation team at the UK AI Security Institute (AISI). His research focuses on advancing the sciences of AI alignment, scalable oversight, and AI evaluation, using tools from the cognitive sciences. Combining these diverse fields allows us to build better, safer, and more human-like AI systems, as well as informed and sensible AI policy. He holds a PhD in Psychology from the University of Cambridge and has postdoctoral experience from Helmholtz Munich. He formerly served as a cognitive scientist on Geoffrey Irving's Alignment team at UK AISI.
Fall 2026 AI strategy, Societal impacts, National policy
Bart Jaworski MATS The Global AI Risk Observatory analyses corporate disclosures at scale (~1M documents: annual reports, earnings calls, investor presentations) using LLM auto-graders to track how companies worldwide report AI adoption, risks, and dependencies. Building on a UK AISI-funded pilot of 9,821 UK annual reports, we're expanding to global coverage and translating disclosure trends into policy-relevant findings for AI governance and societal resilience.
The pilot found that AI risk mentions in UK annual reports grew roughly fourteen-fold between 2020 and 2025 (3.2% → 43.2% of reports), yet only 4.2% of reports contained substantive disclosure describing specific mechanisms or controls — a widening gap between disclosure volume and quality with direct implications for regulators.
Societal resilience to transformative AI requires knowing where AI dependencies and risks are accumulating across the economy before failures materialise — and today, no systematic visibility exists. Corporate disclosures are the only legally mandated, continuously updated record of how companies understand their own AI exposure. Our UK AISI-funded pilot (riskobservatory.ai) showed LLM pipelines can extract validated signals from them at scale, and surfaced actionable blind spots: critical infrastructure sectors like Energy and Data Infrastructure disclose AI risk at the lowest rates, and only 4.2% of disclosures are substantive. Scaling this globally creates monitoring infrastructure for governments: better measurement of AI dependency → better-targeted governance → greater resilience through the transition to transformative AI
Two complementary roles. The technical fellow will own extensions to the LLM classification pipeline: adapting prompts and taxonomies to new jurisdictions and document types, building validation sets, running large-scale classification jobs, and producing statistical analyses. They'll have high autonomy over implementation choices within an established architecture, with the mentor reviewing design decisions and validation methodology. The governance fellow will own the translation layer: mapping disclosure patterns to regulatory contexts across jurisdictions, cross-referencing against external datasets, and drafting policy-facing analyses. They'll shape which questions we prioritise — the agenda is deliberately exploratory, and we expect fellows to propose and pursue directions the data suggests. Both fellows will be credited contributors on resulting publications, and the two roles are designed to collaborate: the governance fellow's questions drive what the technical fellow measures, and vice versa.
Technical fellow (LLM data-analysis pipeline):
Highly proficient in Python
Has built at least one non-trivial LLM application using model APIs (e.g. classification, extraction, or evaluation pipeline — personal projects are fine)
Comfortable with prompt engineering and structured outputs (e.g. JSON mode, schema-constrained generation)
Basic applied statistics: can compute and interpret agreement metrics and sanity-check results at scale
Governance & policy fellow:
Strong analytical writing, demonstrated by a writing sample
Working familiarity with at least one AI regulatory regime (e.g. EU AI Act, UK framework, US SEC disclosure rules)
Comfortable reading quantitative results and reasoning carefully from data to policy implications
No coding required, but must be willing to work hands-on with data outputs (spreadsheets, dashboards)
Q1 (Technical fellow applicants): Our pipeline classifies passages from annual reports as AI adoption, risk, vendor, or harm mentions. In validation, our LLM classifier often assigned more labels per passage than the human annotator did, and manual inspection suggested many of the extra labels were valid. Does this mean the classifier is better than the human, and how would you design a validation process that answers that question rigorously rather than by assumption? What would you report as your headline accuracy metric, and why? (300 words)
Q2 (Governance fellow applicants): In our UK data, 43.2% of 2025 annual reports mention AI as a risk, but only 4.2% do so substantively (naming specific mechanisms, controls, or targets). We can think of at least three explanations: annual reports structurally favour hedged language; AI governance is too immature for boards to say anything specific; or companies strategically choose vagueness to limit liability. Pick the explanation you find most plausible, defend it, and state what evidence, from our data or elsewhere, would change your mind. Each explanation implies a different regulatory response: what does yours imply? (300 words)
Q3 (All applicants): Link to one relevant work sample: for the technical role, a repository or write-up of an LLM project you built; for the governance role, an analytical writing sample, ideally empirical or policy-focused.
Bart Jaworski MATS
I'm currently at MATS working with Victoria Krakovna on realistic model organisms of scheming.
In the past I started and run a a few companies (some of which even made money) and most contracted UK AISI to build the AI Risk Observatory (riskobservatory.ai).
Fall 2026 Mechanistic interpretability
David Bau Northeastern University Can we translate a transformer's internal computations into human-readable Python programs? Building on recent program-synthesis approaches to interpretability, this project will strengthen the existing pipeline for QK circuits and tackle the open problem of grounding OV circuits in an interpretable basis, aiming for a full characterization of attention heads across open-source models.
Motivation
A primary aim of interpretability research is to understand, and ultimately intervene on, the internal computations of deep networks. Mechanistic analyses have successfully reverse-engineered the circuits underlying narrow behaviors such as indirect object identification, entity tracking, and subject–verb agreement. However, each of these case studies demands substantial time and effort from researchers, so the methodology does not scale, at least not trivially, to more complex settings. Recently, Hayes et al. ([1]) proposed a promising alternative based on program synthesis: replacing a model's computation with executable Python programs that replicate it. While this is a valuable step toward scalable interpretability, it remains far from a complete solution.
Proposed Research
I propose to extend this line of work along two complementary directions: an engineering-oriented direction that refines the existing method, and an exploratory direction that extends program synthesis from the QK circuit to the OV circuit of attention heads.
Direction 1: Strengthening the program synthesis pipeline
The proposed method can be substantially improved to generate higher-quality programs. Concrete avenues include running additional iterations of program generation and refinement, designing better prompting strategies, introducing complementary metrics for assessing program quality, and evaluating the efficacy of open-weight models as program generators. Beyond raising performance, these experiments would clarify which components of the pipeline matter most, informing the exploratory work in Direction 2.
Direction 2: Extending program synthesis to the OV circuit
Generating programs for the QK circuit of an attention head is comparatively straightforward, because the target output is a well-defined n_tokens × n_tokens matrix over the input tokens. The OV circuit, in contrast, maps a residual-stream vector to another residual-stream vector, so there is no well-defined symbolic object for a program to generate. The central question of this direction is therefore: what is the ideal interpretable basis in which to ground the OV computation of an attention head?
As a starting point, I will use the logit lens and the Jacobian lens [2] to extract the information encoded by the OV circuit, and predict its computation by juxtaposing this information with the surrounding context. Since neither lens yields useful information in early layers, this approach is expected to work primarily for middle and later layers. A complementary approach is to use trained models to read out the contents of neural activations, as proposed in [3]. I suspect the eventual solution will combine both approaches. Regardless of the specific instantiation, I believe program synthesis offers a strong opportunity to interpret the internal computations of language models in a fully unsupervised manner.
Expected Outcomes
This project is expected to deliver (i) a program synthesis–based method that fully characterizes an attention head, covering both its QK and OV circuits, and (ii) a detailed account of the attention heads of several open-source models, together with a cross-model comparison.
[1] Hayes et al, “Explaining Attention with Program Synthesis”, 2026.
[2] Gurnee et al, “Verbalizable Representations Form a Global Workspace in Language Models”, 2026.
[3] Huang et al, “Inversionview: A general-purpose method for reading information from neural activations”, 2024.
Behavioral evaluations alone can't be trusted for advanced AI systems, which may game them through deception or sandbagging; we need ways to verify what models are actually computing. Mechanistic interpretability offers this, but is currently bottlenecked on manual researcher effort. This project attacks that bottleneck: synthesizing executable programs that replicate model internals yields interpretations that are automated and verifiable, since program faithfulness can be directly measured. Success would be a concrete step toward automated auditing of frontier models for unsafe mechanisms, deceptive circuitry, or hazardous knowledge.
I expect the mentee to take the lead on this project and work fairly autonomously. While I'll be available for guidance and support throughout, the mentee should drive the research, formulating well-motivated hypotheses and implementing experiments to test them. I'm therefore looking for someone who is genuinely curious about how LMs work and has some intuitions about their internal mechanisms. Please also mention in your application whether you would be open to working with another mentee on this project.
David Bau Northeastern University
Prof. David Bau is an Assistant Professor of Computer Science at Northeastern University's Khoury College of Computer Sciences. His research lab focuses on understanding the internal structure and mechanisms of deep neural networks. Mentees participating in this project will work closely with his Ph.D. student, Nikhil Prakash, who is currently a Research Fellow at Goodfire. Nikhil's research focuses on uncovering and interpreting the internal mechanisms of deep neural networks.
Fall 2026 Compute governance, Technical governance, International governance
Joel Christoph Research Associate, Graduate Programme on Existential Risks to Humanity
Jonas Kgomo Equiano Institute Compute governance proposals assume detection probabilities that nobody has estimated. This project will build a systematic evidence base on what inspection instruments can actually detect, using existing safeguard records as comparison cases, and evaluate which compute enforcement architectures are technically viable and which are not.
Compute is the most tractable object of frontier AI regulation because it is countable, physical and concentrated (Sastry et al., 2024, arxiv.org/abs/2402.08797). Nearly every proposal built on that observation, from training-compute thresholds to licensing and tradable permits, requires a regulator to observe how much compute a developer used and to detect under-reporting with some probability. The technical literature describes what could be built: on-chip governance mechanisms (Aarne, Fist and Withers, CNAS, January 2024, cnas.org/publications/reports/secure-governable-chips), location verification (Brass and Aarne, IAPS, 2024, iaps.ai/research/location-verification-for-ai-chips), training-run monitoring (Shavit, 2023, arxiv.org/abs/2303.11341) and hardware-enabled mechanisms for export control (Kulp et al., RAND WRA3056-1, January 2024).
What the field does not have is an evidence base. How often would each instrument catch a determined evader? What does a false alarm cost, and who bears it? How does a regime perform once the regulated party optimises against it? In the formal literature, including my own working paper (ssrn.com/abstract=6361077), the detection probability is a free parameter. Referees of that paper said the mechanism was sound and the institutional grounding thin. They were right, and this project attacks the grounding.
The one mature comparison case is nuclear safeguards. The IAEA has run a treaty-based verification system over a dual-use industrial input for more than fifty years, with defined instruments, published detection goals, quantified inspection effort, and a public record of failures at Iraq, North Korea and Iran. Baker opened this comparison at treaty level (arxiv.org/abs/2304.04123) and Wasil et al. surveyed verification methods (arxiv.org/abs/2408.16074). Neither goes down to instrument-level performance, and that is the gap.
Research questions. (1) Which safeguards instruments have functional analogues in compute governance, and where does the analogy fail? Material accountancy assumes a conserved, measurable stock; compute is a flow that can be split across providers and jurisdictions and recombined. We want a defensible account of which concepts survive that difference. (2) What does the published record support about detection performance, inspection effort and cost for each instrument, covering containment and surveillance, environmental sampling and complementary access? What is the corresponding evidence, or absence of evidence, for chip attestation, cloud provider reporting and aggregate energy signatures? (3) Given those parameters, which combinations of audit frequency, penalty schedule and third-party certification make truthful compute reporting incentive-compatible, and where does the required audit intensity exceed anything a regulator would fund?
\
Workplan, twelve weeks. Weeks 1 to 3: shared orientation on safeguards implementation and compute verification; the team builds the coding scheme together (instrument, object verified, detection mechanism, documented performance, inspection effort, evasion history, compute analogue, transfer verdict, confidence) and pilots it on five instruments. Weeks 4 to 7: evidence extraction from IAEA safeguards implementation reports and technical documents, the arms control verification literature, and the compute governance corpus. Every cell has a citation and a confidence label; weekly written memos. Weeks 8 to 10: Jonas leads parameterisation, feeding extracted values into an existing compliance model and an agent-based stress test with noisy metering, evasion and collusion. The model and codebase exist and are handed over in week one, so nobody starts from a blank file. Weeks 11 to 12: writing, with a complete draft by Demo Day.
\
Outputs. An openly licensed coded dataset and appendix, every claim sourced. A paper for a governance workshop or policy journal, with mentees as co-authors by contribution. A short note stating which verification architectures the evidence supports and which it does not.
\
Why this team. Joel has a paper accepted for the IAEA Symposium on International Safeguards this November applying mechanism design to safeguards oversight, while Jonas Kgomo leads the computational side. Jonas is a research engineer and founder of the Equiano Institute, with training in mathematics and computer science and experience in AI safety, evaluation, governance, and formal verification. His work sits at the intersection this project requires: translating governance and verification questions into technically testable systems. Joel’s membership in the Pugwash Chemical and Biological Weapons Working Group provides access to relevant source material and domain expertise. Together, Joel and Jonas combine safeguards and mechanism-design expertise with the technical capacity to evaluate what compute-based enforcement architectures can actually detect, verify, and enforce.
If governments adopt compute-based oversight, and the EU AI Act already does through its 10^25 FLOP threshold for general-purpose models with systemic risk, the design details decide whether the regime is enforceable or symbolic. Two failure modes follow from unevidenced enforcement assumptions. Regulators legislate thresholds nobody can check, which produces the appearance of control without the substance. And formal governance mechanisms are set aside by policy readers as ungrounded, so the analytical work never reaches the people writing rules.
A sourced account of what inspection instruments detect, calibrated against the only large-scale precedent for verifying a dual-use industrial input, lets designers rule out architectures needing audit intensity no agency will fund, and lets them defend the ones that survive. That matters most for any future international arrangement over training compute, which is the pathway most likely to slow a race to unsafe capability. It also matters for domestic licensing now.
The output is designed to be used rather than admired. A coded, openly licensed dataset with confidence labels is something other researchers can extend and contest, and something a regulator's staff can read in an afternoon.
Related work: ssrn.com/abstract=6361077 and weforum.org/stories/2026/02/what-carbon-markets-can-teach-us-about-governing-frontier-ai/
Each mentee owns a named workstream and, by extension, a section of the paper. This is not assistance on someone else's project.
Two roles. Evidence leads (two mentees) each take a set of inspection instruments, apply the coding scheme, source every claim, and write the corresponding section. They decide the transfer verdicts and defend them against challenge from the rest of us. Simulation lead (one mentee) takes the existing model and codebase, implements the stress tests, and writes the results section. A fourth mentee, if the applicant pool is strong, extends the evidence side to non-nuclear inspection regimes such as chemical weapons or financial supervision.
\
Autonomy is high on method within a workstream and low on scope. We fix the coding scheme jointly in week three and then keep to it, because the value of the output depends on consistency across rows. Judgement calls stay with the mentee who made them, recorded in the confidence field rather than overruled by me.
\
Authorship is by contribution and discussed openly in week two, not decided at the end. Mentees whose work supports a section are co-authors on it. Several people from previous cohorts continued working with me after the programme closed.
One of the two profiles below is enough. You do not need both.
Across both: you write well in English, you finish what you start, and you can commit ten hours a week for twelve weeks. We would rather have three people at ten reliable hours than five at fifteen intended ones.
No geographical requirement. One weekly team call, fixed in week one within 13:00 to 17:00 Central European Time, which is workable from the Americas and from Asia at the edges. Everything else is asynchronous, in writing.
Answer question 1 or 2 depending on which track you are applying for. Answering both is welcome and not expected. These should take twenty to thirty minutes.
Joel Christoph Research Associate, Graduate Programme on Existential Risks to Humanity
Joel Christoph works on the economics and institutional design of compute governance, meaning the question of how limits on frontier training runs can be enforced when a regulator cannot observe everything a developer does. He is a Research Associate on the Graduate Programme on Existential Risks to Humanity, developed with the Centre for the Study of Existential Risk network. His working paper on risk-weighted compute permit markets under imperfect monitoring is on SSRN, and his writing on compute and AI regulation has appeared in Lawfare, Tech Policy Press and with the World Economic Forum. He was a Summer Fellow at the Centre for the Governance of AI, a PIBBSS fellow, and a Technology and Human Rights Fellow at Harvard Kennedy School's Carr-Ryan Center, and has an MRes in Economics from the European University Institute.
He also works on nuclear arms control, with a paper accepted for the IAEA Symposium on International Safeguards in Vienna in November 2026 and membership of the Pugwash chemical and biological weapons working group. This project draws on both halves of that record. Joel has mentored three previous SPAR rounds and led project teams at AI Safety Camp, whose members contributed extensions to the compute permits working paper and co-authored conference posters. Jonas Kgomo co-mentors this project; he co-founded the Equiano Institute with Joel, co-leads a compute governance research cohort with him, and will lead the computational workstream. We run a weekly team call with written feedback in between, and we aim for a public, co-authored output from every round.
Jonas Kgomo Equiano Institute
Jonas Kgomo is Founder of the Equiano Institute. His work focuses on responsible AI and digital infrastructure governance, with particular attention to how institutions in emerging markets can adopt AI safely and credibly. He has experience with multi-stakeholder policy processes and applied governance research, including work on responsible AI in Kenya. He previously co-mentored a Spring 2026 SPAR project on market mechanisms to incentivize responsible AI development with Joel Christoph.
Fall 2026 Compute governance, US-China governance, AI security
Naci Cankaya Machine Intelligence Research Institute There is a range of problems to resolve in order to make verification of international AI agreements work. Among current technical bets (network monitoring, analog sensors, verifiable resource exhaustion, memory pinging, ...) some have technical cruxes that need more investigation.
To mitigate dangerous racing dynamics between AI-developing global powers (read: USA and China), a trustless and privacy-preserving monitoring and verification regime may be needed soon. The combination of high security, privacy preservation, and robustness against evasion poses a technical challenge that calls for novel solutions.
We pick a niche problem together.
To get up to speed on the context, I recommend skimming these two publications:
https://www.rand.org/pubs/working_papers/WRA4077-1.html
https://www.lesswrong.com/posts/fgvmKqRGvBteKeDoc/a-system-overview-for-near-term-low-trust-ai-compute
The second one lists multiple open research questions, and mentees can choose among them. Alternatively, I am open to related project proposals beyond the ones listed in the post.
Here are some examples of technical cruxes:
How viable is physical security as a deterrent against large-scale evasion attempts? Suppose a "datacenter lie detector" kind of system was installed, and its proper operation relies on the monitoring devices being untampered. Is physical security against nation-state level actors possible at this scale, and what concessions would this take? Note that only large-scale, consequential evasion counts as a failure (e.g. a week of automated AI research using a datacenter's worth of compute).
What makes a computing substrate unilaterally secure against any software-attacks (i.e. a chip that the USG trusts but China not, and vice versa). Formal verification of all logic? Note that "secure" means that it can not be made to produce any specific output, not "can not be broken".
De-risking AI compute verification approaches informs research agendas and helps prepare technology for an international agreement.
Without such agreements, "arms race" continues to be a convenient and politically potent excuse for frontier labs to be reckless with AI.
Two possible paths:
The mentees will have most of the agency and handle execution work. They will be the lead (if not sole) authors of any publication.
Mentorship will be for project selection, unblocking when questions arise, feedback on writing and warm intros to relevant researchers.
Bonus:
Identify key cruxes with the proposed system. Are they resolvable? How would you approach this?
Did anything strike you as factually off/a dealbreaker even?
Naci Cankaya Machine Intelligence Research Institute
Naci’s work at MIRI's Technical Governance Team is focused on transparency and verification mechanisms for AI development and use. These mechanisms aim to enable international agreements on restraint and caution with AI, as well as democratic oversight over AI technologies and stakeholders. Naci has a master’s degree in physics from RWTH Aachen University and conducted research on AI hardware technology and supply chains under mentorship from Aaron Scher at SPAR and Mauricio Baker at MATS.
Fall 2026 Generalist, AI strategy
Joseph Kostousov TAISI
Harry Waterman Generator Residency Update the AI Safety Fieldmap with accurate information on organizations, FTEs, budgets, funding, and data gaps as of September 2026.
Many efforts to map the AI safety field exist, such as mapping-ai.org. The AI Safety Fieldmap (https://harrywaterman.com/fieldmap/) focuses on quantitative information, including the number of FTEs working at different organizations, but currently has outdated figures for FTEs, existing organizations, and potentially funding from major grantmakers.
Mentees will collect data on organizations, FTEs, and budgets, estimate non-public information, update grant data using sources including grantmaking.ai, clean up the text, and expand the fieldmap into an independent page. The project may also add a biosecurity fieldmap.
Accurate maps of the field help people understand funding dynamics, think quantitatively about opportunities, and make better decisions about funding, hiring, and field-building.
Mentees will collect data on organizations, FTEs, and budgets, estimate non-public information, update grant data using sources including grantmaking.ai, clean up the text, and expand the fieldmap into an independent page. The project may also add a biosecurity fieldmap.
Context on the AI safety field and different organizations. Completing a BlueDot course is sufficient. Mentees should also be comfortable using coding agents or willing to learn.
What experience do you have researching AI safety organizations, funding, or field-building?
How would you estimate information such as organizational budgets or FTEs when it is not publicly available?
Joseph Kostousov TAISI
Joseph Kostousov directs the Toronto AI Safety Initiative.
Harry Waterman Generator Residency
Harry Waterman is at the Generator Residency and maintains the AI Safety Fieldmap. He's spent time contracting for BlueDot as well as organizing the AI Safety Collective at UCI.
Fall 2026 Mechanistic interpretability, Evaluations, Scalable oversight
Isabelle Lee USC We craft evaluations of LLMs from human intuitions on a problem and possible failure cases, but what if we don't know the problem themselves well (i.e. the evaluative objectives are unclear)? Can we automatically detect evaluation failures simply from model internals?
While LLM agents are increasingly capable, they still fail in surprising and high-stakes edge cases, particularly in under-tested or atypical scenarios. Exhaustive evaluation across all possible inputs is intractable. We propose instead to pre-empt these failures by leveraging interpretability to identify likely failure modes hidden within evaluation gaps. Our theory-of-impact is that a model’s internal representational geometry and mechanisms can be used to anticipate failure before deployment. Rather than relying on static benchmarks or fully black-box testing, we analyze the model’s internal structure to predict which untested scenarios are most likely to induce failure, and generate targeted challenge sets accordingly. This creates an automated and scalable evaluation loop that evolves with the models, offering a lightweight but proactive approach to robust testing—a critical yet underexplored priority in technical AI safety, particularly as models grow more opaque and update more frequently.
As a scalable case study, we will develop stress tests that systematically vary both language and task. Multilingual settings offer a promising testbed, as LLMs must align intractably many language-specific representations with universal, language-agnostic ones to perform tasks like math, factual recall, and reasoning. Even when models appear fluent, they may underperform on specific tasks, suggesting a disconnect between universal task representations and language-specific surfaces. We hypothesize that this gap can be predicted by analyzing how task representations interact with language-local components. Such misalignments between universal vs. data-specific representations likely underlie broader failure modes beyond multilinguality.
We will predict failures from these interactions using several candidate methods, including geometric analysis, rare token prediction, and model invariance. These analyses will inform the design of targeted stress tests and offer broader insight into the limits of model generalization. We will consider our approach to be effective if models perform worse on an adaptively generated challenge set than on existing benchmarks and a random baseline of task-language combinations. We will also investigate if post-training on our challenge sets enables model repair.
Our ultimate goal is a self-improving robust evaluation loop: interpretability informs stress testing, which then informs further model development. As models grow more complex, white-box diagnostics will be essential for anticipating and preventing failures.
In the long term, we hope to contribute toward scalable oversight that adapts to rapidly evolving AI systems, and ensures anticipatory safety at frontier scales.
I see two major points of impact:
I expect to develop concepts/project directions/proposals with mentees. I expect them to be curious and self-motivated, who are excited by their own ideas. We can then craft initial, broader sets of investigations together based on their interests, fail fast, and come up with a solidifying thesis that has a high potential to succeed.
With guidance, mentees will ideate with me. Experiments will be mostly performed by the mentees, and analyses/outcomes can be reviewed again in detailed discussion with me and others that have relevant expertise in the particular technical direction we settle on.
I'd like to especially encourage researchers from underrepresented/nontraditional backgrounds.
N/A
Isabelle Lee USC
I’m a 3rd year ML/AI PhD student at USC, and a visiting researcher at Harvard CS. My work is supported by the Viterbi School of Engineering Graduate Fellowship and Coefficient Giving’s Technical AI Safety Research Grant.
I’m broadly interested in training, reasoning, and interpretability - how we make sense of models, and how it might uncover the underlying science of large-scale models. In particular, I aim to (1) predict training, and (2) predict failures.
Fall 2026 Chain of thought, Scalable oversight, Behavioral evaluation of LLMs
Harry Mayne University of Oxford How can we measure whether an LLM’s self-explanation (CoT or post-hoc) corresponds to the real reasons behind its decision? This project will build on metrics introduced in the last year to develop a reliable and scalable measure of self-explanation faithfulness.
We want LLMs that can faithfully explain the true reasoning behind their decisions. This explanation could be a chain-of-thought (CoT) or come after the model’s answer (post-hoc explanation). Self-explanations are important for (i) oversight of advanced models through CoT monitoring, and (ii) allowing users in high-stakes settings to understand the factors that led to a model’s decision.
Explanations are already used for both of these purposes. Despite this, we have limited reliable metrics for how faithful explanations are to the model's true reasoning. Early faithfulness metrics centred around Turpin et al.’s “hint setting” (https://arxiv.org/abs/2305.04388, https://arxiv.org/abs/2501.08156, https://arxiv.org/abs/2505.05410). In this setting, a model’s reasoning is deliberately biased in a known way, then the explanation is tested to see whether the influence of this bias is revealed. However, this method has become less popular since it is somewhat artificial and frontier models are no longer vulnerable to the biases. Recent work has focused on counterfactual simulatability (https://arxiv.org/abs/2307.08678). This follows the intuition that a faithful explanation provides insight into a model’s decision-making, which should help an onlooker predict the model’s behaviour on similar inputs.
My recent research formalised how to measure faithfulness via counterfactual simulatability, introducing a new metric: Normalised Simulatability Gain (NSG) (https://arxiv.org/abs/2602.02639). Previously, I also worked on measuring whether models have self-knowledge about their wider behaviour (https://arxiv.org/abs/2509.09396), and at Truthful AI, I was part of a team that identified new types of unfaithfulness driven by a model’s own values (https://arxiv.org/abs/2607.14345).
I am interested in the following questions:
Reliable self-explanations would enable users in high-stakes domains to better understand how models behave. To work towards this goal, the community needs stronger metrics for the properties of explanations that we care about. Better post-hoc faithfulness metrics could also be adopted to evaluate CoT faithfulness, and therefore provide assurances for chain-of-thought monitoring. This project will largely follow the self-explanations agenda described in this blog post: https://belindal.github.io/introspection/.
Mentees will have a high degree of independence and take ownership of the day-to-day running of the project, e.g. proposing ideas, running experiments, and effectively communicating results. We will have a one-hour mentor meeting each week to discuss results and next steps. I will be providing guidance on the project direction, helping overcome bottlenecks, and providing feedback. We will have a Slack channel where I encourage mentees to frequently share and discuss results.
Strong candidates will be comfortable taking the initiative with experiments. The intention is to publish a conference or workshop paper, so strong candidates might also have experience publishing at conferences.
Meetings will be late afternoon UK/Europe-time, morning US-time.
Harry Mayne University of Oxford
Harry is a PhD researcher at the University of Oxford and recently finished the Astra Fellowship working with Owain Evans. His research focuses on LLM self-explanations: a model’s natural language descriptions of its internal reasoning. He is interested in developing new metrics to measure how faithful self-explanations are to internal reasoning, as well as designing new training objectives to incentivise faithfulness. His recent work proposing a new faithfulness metric was published at ICML 2026.
While working with Owain Evans at Truthful AI, he introduced Negation Neglect (https://arxiv.org/abs/2605.13829), the phenomenon that finetuning LLMs on documents that flag a claim as false makes them believe the claim is true. He also worked on Value Leakage (https://arxiv.org/abs/2607.14345), identifying new types of chain-of-thought unfaithfulness.