Monitoring and Attributing Implicit Personalization in Conversational Agents

When language models interact with users, they implicitly infer user attributes (e.g., expertise level, demographics, beliefs) that influence their responses in ways users often neither expect nor endorse. While previous work in the implicit personalization domain focused on detecting and controlling a restricted set of demographic attributes (see: https://aclanthology.org/2025.emnlp-main.1029/, https://arxiv.org/abs/2406.07882), recent work demonstrated that more complex user models can be verbalized from model activations using trained decoder probes. This project aims to build upon these findings to expand our understanding of implicit personalization in LLMs, asking questions such as:

• Are latent representation of user attributes uniform across model layers? Do different model families (e.g., Llama, Qwen, Gemma) converge towards similar user representations, or are certain systems more prone to problematic user modeling?
• How do user models evolve across multi-turn conversations? Which conversational elements most strongly drive implicit personalization, and do these match users' expectations about what information they are (implicitly) revealing? Can we trace these behaviors back to specific training examples in which characteristics are expressed more explicitly?
• How can advanced implicit personalization probes be incorporated within interfaces for live monitoring of user models, such as TalkTuner? Can we scale these workflows to larger models exploiting NNSight for remote execution?
• Which resources can be created to study implicit personalization in multi-turn realistic settings, focusing in particular on subtle cues rather than explicit system-prompt signals? How can we leverage synthetic personas to generate such resources at scale?
• How related are

Depending on mentee background and interests, we will prioritize 1-2 of these directions for the project duration.

Key Readings

• Scalably Extract Latent Representations of Users (Choi et al. 2025, Transluce Blog)
• Implicit Personalization in Language Models: A Systematic Study (Jin et al. 2024, Findings of EMNLP)
• Towards Understanding Sycophancy in Language Models (Sharma et al. 2023, Anthropic Blog)
• Reading Between the Prompts: How Stereotypes Shape LLM’s Implicit Personalization (Neplenbroek et al. 2025, EMNLP)
• Finetuning LLMs for Human Behavior Prediction in Social Science Experiments (Kolluri et al. 2025, EMNLP)
• Generative Agent Simulations of 1,000 People (Park et al. 2024, Arxiv)
• Designing a Dashboard for Transparency and Control of Conversational AI (Chen et al. 2024, ActInterp Workshop @ ICML)
• Creating general user models from computer use (Shaikh et al. 2025, ACM UIST Hon. Best Paper)
• On the Biology of Language Models (Lindsey et al. 2025, Anthropic Circuits Thread)

User models investigated in this project are a key mediator of concerning behaviors such as sycophancy, deception, demographic bias and reward hacking. For example, models that infer user beliefs may prioritize agreement with their interlocutor over answer accuracy, or provide lower-quality responses to users perceived as unlikely to verify correctness. As models become more capable, these behaviors become harder to detect through behavioral testing alone, and interpretability tools prove useful to surface latent user models and causally connect them to problematic behaviors. In particular, this project aims to contribute to AI safety by: (1) developing methods to detect when models are engaging in potentially problematic personalization, (2) characterizing which model architectures and training approaches are more prone to these behaviors, and (3) creating resources (datasets, interfaces) that enable the broader research community to further study implicit personalization.

Mentees will take a leading role in defining the research questions within the project's scope, and refining them in light of early findings. They will be responsible for reviewing relevant literature, conducting the necessary technical work, including adapting existing codebases, training decoders, or building evaluation pipelines. Weekly check-ins will provide guidance, but mentees should expect to work independently between meetings, debugging issues and iterating on experimental designs. They will also be responsible for the redaction of the mid-term and final reports.

• Strong programming skills in Python
• Some experience with PyTorch, including working with model internals (hooks, activations, gradients). Familiarity with HuggingFace Transformers is strongly preferred.
• Good understanding of transformer architectures, fine-tuning, and evaluation methodology. Coursework or project experience equivalent to a graduate ML course is a plus.
• Some familiarity with mechanistic interpretability concepts (probing, activation patching, circuit analysis) the NNsight interpretability toolkit are a plus.
• Prior research experience (course projects, internships, or independent work) preferred. Prior reading of relevant papers is expected; hands-on experience is a plus but not required.
• Prior experience in cognitive modeling, or work with behavioral signals or neural recordings is a plus.

10-20 hrs/week depending on team size

Mentees from any region are welcome to apply. Ideally, they’d be available for meetings during EST hours, preferably on Saturdays, but scheduling is flexible.