When a model fails a task in an evaluation, we record it as evidence the model lacks the capability. However, sometimes, the evaluation is what failed: a broken tool, a truncated context, an unparsed answer, an exhausted step budget, a refusal, or a mistaken grader. This project builds a taxonomy of these evaluation failure modes, a suite of automatically computable indicators that detect them from run traces, and a prescribed mitigation for each, so that "the model cannot do this" can be distinguished from "our harness did not let it."
About the project
Adversarial ML has long history of misreading failed attacks as evidence of robustness. A weak attack that failed for optimization or other unrevealed reasons — vanishing gradients, poor initialization, a badly chosen loss, too few iterations — produced the same observable as a genuinely robust model, and many defenses were published on that basis and later broken. This is why I focused my PhD in developing indicators of attack failure: cheap diagnostics computed over the attack's optimization trace, each tied to a specific fix, letting practicioners tell a failed attack from a robust model before publishing.
LLM and agent evaluations are now in a similar position. A zero on a task gets reported as a capability limit, but it may be a harness artifact or a setup issue instead, and little effort goes into auditing it. This matters most for dangerous-capability evaluations, where a reported score is only a lower bound on true capability and evaluation failures widen the gap between that bound and reality.
Research questions:
- What are the recurring causes of failed runs in agent and reasoning evaluations, and what fraction of recorded failures are evaluation artifacts rather than model failures?
- Which properties, computable automatically from a run trace, reliably flag each cause?
- Does applying the prescribed mitigation actually recover score, and is the recovered gap large enough to change benchmark rankings or safety conclusions?
The study will start by running traces from a few public agentic and reasoning benchmarks and hand-label a stratified sample of failed episodes by root cause. The failures will be categorized mostly by how they are detectable and how they can be mitigated. Tentative categories for now could include: harness and I/O failures (parse errors, format violations, malformed tool calls, sandbox errors); budget exhaustion (step, token, or context limits hit while the trajectory was still progressing); elicitation failures (no chain of thought, underspecified prompts, undocumented tools, single-sample evaluation of tasks that need retries); behavioral failures that aren't capability failures (refusals, abstention, degenerate loops); grading failures (wrong ground truth, over-strict exact match, judge miscalibration, tasks with multiple valid solutions); task specification failures (ambiguous goals, environments broken by dependency drift); and the inverse case of spurious success from reward hacking or contamination.
After the first categorization, the project will develop measurable indicators from the traces. Finally, we will validate the method by testing a few application use cases including at least one LLM benchmark and one agentic benchmark, detecting causes of failures of attacks and applying the mitigation to runs whose indicators fired should recover score, while applying it to clean runs shouldn't. Measuring the recovered gap per benchmark and model, and checking indicator precision and recall against the hand labels will demonstrate the utility of the indicators.
References: Pintor et al., "Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples," NeurIPS 2022. Carlini et al., "On Evaluating Adversarial Robustness," 2019.
Theory of change
This project will lead to debugging tools to evaluate security and safety of LLMs and AI agents. This project might lead to re-evaluating existing research in these topics with a new lens that goes beyond measuring success of the attacks and investigates true worst-case evaluation.
Your role
Mentees will conduct research and help shaping the idea and outcomes. They will concretely participate by: collecting run traces from the benchmarks, inspecting and labeling failed episodes by root cause, and proposing revisions to the failure taxonomy based on what they actually see in the transcripts. My candidate categories are a starting point, not a fixed schema, and I expect the labeling to change them.
From there, mentees will implement the indicators, run the mitigation experiments, and analyze the results. I'd expect to set direction and priorities in weekly meetings and to be closely involved in the first two weeks while the labeling protocol is being worked out, then step back to reviewing results and unblocking as they take over day-to-day decisions about which benchmarks to pursue, which indicators are worth implementing, and how to structure the experiments.
If two mentees join, they can collaborate with me in the methodology design and requirements, whereas they will split their work on the two proposed use cases of LLM and agentic security and safety testing.
I'm looking for someone who will push back when the taxonomy doesn't fit what they're reading, rather than labeling to my categories.
Prerequisites
Comfortable with Python and running agents at moderate scale, and some prior experience with AI security and safety is appreciated. Willing to read a great many transcripts. The labeling phase is unglamorous and is where the insight comes from.
Location preference
Europe
Application question(s)
- Please, provide a critique on the use of LLM as a judge
- Which tools would you use to better assess and debug the evaluations of LLM and agentic systems?
- Suppose an eval reports that a model solves 40% of tasks. You raise the step budget, add a retry-on-parse-failure loop, and switch from exact-match to an LLM judge, and the score rises to 55%. Which of these three changes, if any, do you consider a legitimate fix rather than gaming the eval, and why? Is there a principled line here, or only case-by-case judgment?
- Name one failure mode you'd expect to be common in agent evals that would be hard to detect automatically from a run trace, and explain what makes it hard.
About the mentor
I am an Assistant Professor at the PRA Lab in the Department of Electrical and Electronic Engineering at the University of Cagliari, Italy. I received my PhD in Electronic and Computer Engineering from the University of Cagliari in 2022.
My research focuses on adversarial machine learning, with a particular emphasis on evaluating, debugging, and improving the robustness of machine learning systems. My work aims to make AI models more reliable, secure, and trustworthy in real-world settings.