Automated Circuit Analysis for Real-time Internal Monitoring

What is this project about?

Circuits [1,4, 6] are one of the best tools we have to get an insight into LLM internals. Open-source tools like circuit_tracer [5] can automatically construct circuits for any given prompt (see Neuronpedia [3] for instance). However, there are still a couple of issues:

1. The raw circuits are still highly information dense with many nodes and edges between them. The nodes need to be clustered into interpretable supernodes before it can be understood properly.

2. The circuits are typically specific to a single prompt. Understanding how an LLM completes a task would require a dataset-level analysis. However, manually analyzing circuits for each prompt in a dataset would be too time-consuming. Some semi-automated methods exist in the literature [2] but they are inherently limited in scope.

The solution (as I envision it) is an agent which can take these circuits as input and produce an actionable insight as output. This could be a probe to detect future undesirable actions, or an intervention that can surgically change a target behavior, or something else.

How do we plan to go about this?

Keeping in mind time/resource constraints, the plan is to answer the following research questions in order:

1. **Can we get a state-of-the-art LLM (like Gemini 3) to produce sensible analyses of circuits? ** I expect the answer is "yes", but we would need to think a bit on how the circuit is presented to the model. The ideal format should be a high-level summary first, with tools (provided to the model) that can provide more information if desired.

2. **Can we get the LLM agent to replicate prior human analyses? ** Next, we should check if we can get the agent to produce analyses similar to human ones, such as Marks et al 's [2] spurious feature detection.

3. **Can we get the LLM agent to perform novel analyses? ** Now, we want to analyze more complex phenomena that can really prove the superiority of agentic interpretability. Good candidates include refusal, scheming, and internal planning.

** Goal and Deliverables **

At a minimum, we should be able to put out an open-source Github repository + blogpost for others to use and build on top of. With decent results, we should also be able to target conferences such as ICLR/NeurIPS/ICML.

Preparation

Apart from the papers/links already cited, please also familiarise yourself with:

• Mathematical Frameworks for Transformer Circuits (https://transformer-circuits.pub/2021/framework/index.html)
• Auditing Agents (https://alignment.anthropic.com/2025/automated-auditing/)

Prioritize the cited references, though.

References:

[1] Anthropic, On the Biology of a Large Language Model, https://transformer-circuits.pub/2025/attribution-graphs/biology.html

[2] Marks et al, Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, https://arxiv.org/abs/2403.19647

[3] Neuronpedia Gemma2 2B circuits, https://www.neuronpedia.org/gemma-2-2b/graph

[4] Dunefsky et al., Transcoders Find Interpretable LLM Feature Circuits, https://arxiv.org/abs/2406.11944

[5] Circuit Tracer, https://github.com/safety-research/circuit-tracer

[6] Anthropic, Circuit Tracing: Revealing Computational Graphs in Language Models, https://transformer-circuits.pub/2025/attribution-graphs/methods.html

[7] Mathematical Frameworks for Transformer Circuits (https://transformer-circuits.pub/2021/framework/index.html)

[8] Auditing Agents (https://alignment.anthropic.com/2025/automated-auditing/)

1. An agent like the one described in the proposal would make significant headway in automatically monitoring/auditing the internals of deployed agents. Coupled with CoT monitoring, this could be a comprehensive solution for monitoring AI agents.

2. In the near term, it may help interpretability researchers to conduct more large scale analysis of circuits and uncover common themes and patterns in these circuits, enabling the discovery of more powerful high-level abstractions.

Mentees are allowed full autonomy inside the overall plan outlined in the proposal, and will be in-charge of executing the project. However, I can always suggest concrete next steps and actions if desired (basically what I would do if I were leading the project).

In case we aim for a conference publication, I can lead the writing fully, leaving the mentees to focus on the experiments.

If the group is larger, it would be better if the roles are more clearly demarcated early on, so that the division of responsibilities is clear.

• Highly proficient in Python
• Expertise in data structures
• Familiar with agentic frameworks (DsPy is good: https://dspy.ai)
• Know how asynchronous API calling works.
• Familiar with the cited works (or can quickly read and understand them before/during the first week)
• Likes playing around with circuits!