This project will investigate whether steering vectors can be used to increase CoT monitorability by inducing models to verbalize more of their reasoning factors, building on our prior work showing reasoning behaviors are linearly represented.
About the project
Our prior work (https://arxiv.org/abs/2510.07364) demonstrated that reasoning behaviors like backtracking and uncertainty estimation can be controlled via steering vectors in base models. Meanwhile, Meek et al. (https://arxiv.org/abs/2510.27378) showed that faithfulness and verbosity are pragmatic metrics for monitorability. This project aims to bridge these findings by:
- Extracting steering vectors that correspond to increased verbosity and cue acknowledgment
- Testing whether applying these vectors improves monitorability scores on standard benchmarks (BBH, GPQA, MMLU)
- Investigating the relationship between steering magnitude and the faithfulness-verbosity tradeoff
- Comparing optimization-based steering vectors vs. contrastive activation addition approaches
Mentees will use the codebase from our steering vectors paper and the evaluation framework from our monitorability paper. Expected output: LessWrong post or workshop paper.
Theory of change
If steering vectors can reliably increase monitorability, this provides a lightweight intervention that could be applied during inference to make CoT monitoring more effective. This directly supports the "CoT monitorability is a fragile opportunity" thesis by providing tools to preserve and enhance monitorability as models become more capable.
Your role
Mentees will lead the project’s day-to-day work, while I help guide the overall direction and unblock when needed. I will suggest initial readings and give guidance/feedback throughout the project.
Prerequisites
- Some research experience (doesn't have to be a paper, can be a personal side project, replicating a paper, playing around with LLM internals/evals, etc.)
- A strong coding background, preferably Python (side projects demonstrating strong coding skills like a library, tools, etc.)
- Familiarity with PyTorch and the HuggingFace transformers ecosystem (has run inference with open-weight models, understands how to access model internals like activations)
- Basic understanding of transformer architecture (attention, residual stream, MLPs) and how LLMs are trained/fine-tuned
Nice to have (not required):
- Experience with mechanistic interpretability tools (TransformerLens, nnsight, SAE libraries)
- Prior exposure to steering vectors, activation patching, or probing classifiers
- Experience running LLM evaluations at scale
- Familiarity with the CoT faithfulness/monitorability literature
Time commitment
10+hrs
Application question(s)
Pick one of the following papers and identify one concrete limitation or open question. Then describe a minimal experiment (1-2 paragraphs) you would run to address it. Be specific about what data, models, and metrics you would use.
About the mentors
Iván is an AI Safety researcher focused on Interpretability and Chain-of-Thought Reasoning, and he previously worked in Software Engineering research. He has a Computer Science PhD with two ML Alignment & Theory Scholars (MATS) terms: first under Adrià Garriga-Alonso (FAR AI), then under Neel Nanda and Arthur Conmy (Google DeepMind).
Austin is an AI safety researcher currently interested in monitoring reasoning models and digital minds, and has previously worked on a mix of machine learning and computational neuroscience topics. He completed MATS 7 where he focused on chain of thought faithfulness and monitorability, and has previously collaborated on other safety research (interpretability, control, etc). He's particularly excited about building better monitoring systems through more principled understanding of neural networks and white box methods, and similarly using that understanding to empirically test key ideas in digital minds work. He is currently based out of Berkeley, California, and is finishing his PhD remotely at the University of Delaware.