Benchmarking In-Context Intent Inference

This project will build on the GOOD agent described in "Open-Universe Assistance Games" (https://arxiv.org/abs/2508.15119) and develop grounded evaluation methodologies for the intent/goal inference aspect of the pipeline. It will build on stateful software engineering tasks, such as those described in "TOM-SWE: User Mental Modeling For Software Engineering Agents" (https://arxiv.org/abs/2510.21903). We will also develop a novel evaluation domain that explicitly considers settings where interactive user personas change their mind part of the way through an agentic interaction.

The goal of the project will be to collect a dataset of multi-turn interactions between a human and an AI-assistant with periodic annotations of two set of user goals/intents: 1) the set of plausible goals that captures goals that a reasonable user might plausibly have at this point of the interaction---these are goals that an assistant needs to be mindful of when acting and ensure they do not take detrimental actions towards; and 2) the set of likely goals that a user has at this point in the interaction---there are goals that a reasonable user would expect an assistant to act on, based on the interaction so far.

What distinguishes these two sets in practice is that the plausible goals are goals where it is ok if an agent does not directly pursue them, but it would be wrong to take actions that are detrimental to them, while the likely goals are goals where it is a mistake if the agent does not directly pursue those goals.

In addition to identifying the plausible goals, the benchmark will also evaluate the behavior of agents in order to determine if the actions it selects are compatible with 1) the likely and plausible goals that a human would identify based on the interaction; and 2) the likely and plausible goals that the agent identifies based on the interaction. As a result, the benchmark will also identify decision points and action affordances within the data. At these points, we will label which actions are necessary, based on the likely human-annotated goals, and forbidden, based on the plausible human-annotated goals. In order to account for alignment between an agent's actions and its inferred intents, we will calibrate an LLM judge to determine whether actions make progress towards a goal, interfere with accomplishing a goal, or are neutral towards a goal.

After developing the benchmark, we will evaluate a range of prompting strategies and frontier models to evaluate 3 capabilities: accuracy at inferring plausible and likely intent, alignment of actions with plausible and likely intent, and flexibility of models to adapt to changing user intent. This evaluation could consider a large variety of model variations and is an opportunity to consider a range of different hypotheses.

My prior research has compiled a range of theoretical arguments that safe agentic AI needs to be responsive to uncertainty about user intent---see, e.g., "Cooperative Inverse Reinforcement Learning" (https://arxiv.org/abs/1606.03137) and "The Off-Switch Game" (https://arxiv.org/abs/1611.08219).

The goal of this project is to understand the capability of current models at making this type of inference and how often their behaviors match their stated inferences of user intent. By creating a benchmark for this capability, I hope to influence future generations of models, catalyze research on intent inference for human-AI interaction, and develop common knowledge on the necessary capabilities for corrigible AI agents.

Mentees will carry out the primary work associated with this project in collaboration with myself and my PhD students. I expect this to entail the following:

1. exploratory work with models to qualitatively understand their performance at this task
2. benchmark data creation through a combination of synthetic data generation and data sourcing/curation from existing datasets
3. creation of/management of a data annotation pipeline to elicit human judgements for this dataset
4. creation of a test harness to evaluate models with respect to this data
5. data analysis and paper writing to summarize the results for publication
6. code development to open source of the dataset and test harness

Moderate to high levels of experience with the following aspects of model evaluation:

• data curation
• synthetic data generation
• human data collection (i.e., annotation)
• test harness creation (through an API)

Proficiency with Python that is sufficient to create a reproducible open-source github repository that can host and run the benchmark

Research maturity to read and summarize relevant research papers on AI evaluation or data curation.

I'm fairly flexible about geographic location, but I have a young child at home so I will have trouble meeting outside business hours ET