FragGuard aims to detect malicious model misuse for cyber where queries are decomposed into multiple sessions to mask the malicious intent.
About the project
LLM-driven code generation APIs face a critical security challenge: malicious requests can be decomposed into multiple benign-appearing fragments that individually pass safety filters but collectively implement harmful capabilities like reverse shells, keyloggers, or data exfiltration. This fragmentation attack, first described by Anthropic's report of cyber espionage in November 2025, exploits the fundamental limitation that maliciousness is not a local property. Dangerous intent emerges only through multi-step composition across sessions.
In this project, we aim to address this limitation with FragGuard, a global detection system that analyzes cross-session semantic patterns. Rather than classifying individual code snippets, FragGuard constructs directed graphs capturing dependencies between fragments (data flow, temporal ordering, shared resources) and applies Graph Attention Networks to identify distributed attack chains. We train the model adversarially by automatically fragmenting and obfuscating sample kill-chains, enabling robust detection of evasive attacks.
Theory of change
Our project should reduce the risk of malicious actors using model APIs to generate carry out cyberattacks. We intend to expand our threat model to include a model leveraging cyber as an escape vector in loss of control scenarios as well.
Your role
Mentees will be assigned tasks by mentors and work on them more or less independently, but mentors will be available on slack and through scheduled calls. Mentees will be expected to write code and run experiments, but not necessarily to write LaTeX unless they want to. Mentors may ask to review pull requests depending on the experience of mentees.
Prerequisites
- Highly proficient with Python and git (or Github).
- Has written code to run transformers from Huggingface, with custom system prompts.
- Has run code on GPUs from the command line on a cluster (or a local machine).
Location preference
We will probably aim to hold meetings between 1pm GMT and 9pm GMT most of the time.
Application question(s)
In your opinion, how long will it be until nearly all coding tasks can be automated by AI? Why do you say this timeline? Optional reading: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ (200 words)
Why would it have been impossible for any individual model safeguards to prevent the cyberattack described in this paper? What might have worked instead? https://www.anthropic.com/news/disrupting-AI-espionage (200 words)
What is an example of a typical failure of a coding assistant like Claude Code? How can an agentic framework mitigate this failure? (200 words)
About the mentors
Linh is an AI safety researcher at Mila, where she has worked on alignment through latent adversarial training for personalities. Linh participated in the MARS research program with Geodesic Research, working on chain of thought monitorability. She previously did a postdoc at the University of Technology Sydney and obtained her PhD from the University of Queensland in natural language processing.
David enjoys AI safety research and mentorship, and currently works full-time as a research manager at ERA. David was an early member of Yoshua Bengio's LawZero in Montreal, and has also written grants and conducted research at Mila. He participated in the MARS program with Geodesic Research, working on chain of thought monitorability. David previously spent four years as a cyber insurance startup CTO, leading a 15+ person team. He holds a cybersecurity PhD from Columbia University. David also works in AI risk communications, with a 30,000+ subscriber YouTube channel.