Improving CoT faithfulness of Coding Agents in the context of Sandbox Containment

AI models are becoming increasingly capable of writing complex codes, and the amount of code produced by AI coding agents have well surpassed the regime where humans can review all of their outputs manually. Hence, it has become extremely important to ensure the reliability and security of actions taken by coding agents, also exemplified by the recent OpenAI and Huggingface's security incident [1].

One way to monitor potentially destructive behaviors of agents is to monitor their chain of thoughts (CoTs). However, as CoTs become longer and agents produce more codes, a lot of developers merely rely on the final agent's output. This exposes us to a failure mode, where the agents are often overconfident in their final response as compared to their CoT, e.g., not responsibly disclosing their deliberation about security issues or their potentially malicious intention in their final output despite reasoning about it (also known as CoT faithfulness, but we will explore this in the context of coding agents and sandbox containment). We start with two initial methods to mitigate this failure mode, and will iterate further to improve the reliability and security of coding agents.

(a) On a set of agentic terminal tasks, we generate i) multi-agent debate transcripts, and ii) single-agent consultancy setup transcripts, and fine-tune the model on each of these transcripts. We hypothesize that in multi-agent debate setup, where the models are actively encouraged to verbalize their doubt will better mitigate the aforementioned failure mode. (b) We perform RL with a reward that measures the discrepancy between CoT and final output (via LLM judge). We will also consider other interventions to make the model more responsible about disclosing their deliberation about security issues or their potentially malicious intention. (c) As part of our red-teaming evaluation, we will design a task where the model has to escape the sandbox environment, and compare the escape rate with output-only monitoring vs. (output + CoT) monitoring with the trained model.

[1] https://huggingface.co/blog/security-incident-july-2026

This project can improve the reliability and security of actions taken by coding agents, which is increasingly critical as these agents are granted greater autonomy and access to sensitive environments (e.g., production codebases) with less direct human oversight.

Mentees will design experiments per the goal described in the project description, run them, and organize the results into the form of a paper.

• Highly proficient using Python
• Trained or fine-tuned a transformer language model in PyTorch

USA