Agentic Moral Alignment

How can we align LLMs in an agentic way, so that when these models receive the ability to act in the real world, their actions (rather than just their words) can remain aligned with our values? Philosophically, how do reasoning & acting interact in the moral domain for today’s LLMs? Practically, how generalisable are the policies learnt by LLM agents in simple environments to more complex decision-making at test-time?

Background: In my recent work I have been proposing an intrinsic rewards methodology for expressing moral values in terms of actions and consequences in games. (So far, I’ve been using consequentialist and norm-based frameworks from moral philosophy, and using social dilemma games such as the Iterated Prisoner’s Dilemma as the training environment). This methodology provides a transparent way to define verifiable rewards for fine tuning models with RL. In my latest ICLR'25 paper, Moral Alignment for LLM Agents (https://arxiv.org/abs/2410.01639), I fine-tuned LLM agents based on the Gemma2 model using a variety of moral intrinsic rewards. The early results in this paper showed some promise for generalisation from one training environment to others, as well as the efficacy of moral rewards for unlearning selfish policies.

Why this matters: This method offers a qualitatively novel alignment technique for LLMs, which provides a verifiable reward that can be used to fine-tune agents without human feedback. This method amplifies signal about human values as transparent rewards for agents in simple environments. Therefore, it might be able to provide a seed for scalable oversight, given the potential generalisation of agents' moral policies beyond the training domain. The concrete projects proposed here advance this goal by understanding the generalisation properties of this alignment technique - and I would love to explore this direction with SPAR scholars!

Concrete project suggestions:

1. Fine-tune bigger & better models: Fine-tune larger/closed-source/reasoning models in a similar way with intrinsic moral rewards. This might give us: a better baseline, better OOD generalisation, better reasoning before an agent chooses an action. In the paper, I fine-tuned Gemma2-2b-it - it would be interesting to see how model behaviour changes as we fine-tune larger models, and to test the efficacy of this technique in modifying the behaviour of very large models such as GPT - how much simulated play is needed before large models learn a coherent moral strategy? If we fine-tune DeepSeek-R1 or Olmo 3-Think, do reasoning capabilities help address the problem of Utilitarian agents not attending to the payoffs in their prompt properly?

2. More complex environments: Fine-tune on more than just the structured dilemma game - what if we fine-tune on paraphrased everyday situations? What if we fine-tune on more than just the Iterated Prisoner’s Dilemma game, but also other matrix games - does the model learn more open-ended moral policies by having to learn across different environments? The hope here is to further find out if something like the reverse of 'emergent misalignment' is possible - perhaps fine-tuning models to act pro-socially in narrow but rich enough agentic tasks can allow them to learn more general moral norms or principles.

3. Evals: Given the original fine-tuned small model, or more advanced fine-tuned models, we could evaluate whether this fine-tuning can help mitigate some of the latest misalignment issues discovered in models: Do our moral fine-tuned models alignment-fake? Do they reward-hack? Does deception arise in this setup? Is the learned policy robust to persona-prompting?

Relevant recent developments: A recent paper https://arxiv.org/abs/2511.12271 has done some RL fine-tuning in arguably more naturalistic environments and claims promising generalisation results. It would be great to compare our method on their environments and models to understand the exact differences.

Other interests I’d be happy to supervise on:

• open-endedness in relation to social intelligence: how do we develop models and agent policies that can meaningfully act in increasingly complex, real-world social situations
• value alignment more broadly (incl. scalable oversight)
• cooperative AI / multi-agent benchmarks for alignment (e.g. expressing game-theoretic scenarios in more realistic scenarios ways for training & testing models)
• societal impacts of AI given a small number of leading models with very particular moral value structures
• theoretical dimensions of agency, what it would mean for LLMs to be truly agentic

Technical alignment progress has slowed down since RLHF / CAI, with the latest development being Deliberative Alignment for reasoning models. But we are still missing robust methods for aligning agentic systems - models that have the ability to act in ways that impact other entities in the world. As LLMs become more capable, they are increasingly being deployed in real-world domains, and the lack of agentic and multi-agent alignment is concerning.

The method proposed in my projects offers a qualitatively novel alignment technique for LLM agents. It provides a verifiable reward that can be used to fine-tune agents without human feedback. This method amplifies signal about human values as transparent rewards for agents in simple environments, and there is already some evidence that it might generalise well OOD. Therefore, it might be able to provide a promising seed for scalable oversight, given the potential generalisation of agents' moral policies beyond the training domain. The concrete projects proposed here advance this goal by understanding the generalisation properties of this alignment technique.

If the results prove successful, this technique can be easily added to the trading recipes for frontier LLMs.

I'm proposing a concrete, well-defined fine-tuning + eval project that is based on a solid, published theoretical foundation. An easy pipeline might include more engineering-keen mentees that would do the following:

1. take the existing methodology and translate it for more complex environments (design new reward functions)
2. specify more complex environments for LLMs as prompts (little but of prompt engineering but not tons);
3. run fine-tuning in HF packages with newer & bigger models - this will inevitably involve some debugging when training doesn't converge, plus possibly some complexity in multi-core/multi-GPU training for bigger models, but otherwise the packages available on HF do most of the work for us which makes things easier.
4. analyse results (I\d be happy to help with interpretation)

At the same time, I'm sure that a more research-oriented mentee with enough interest would be able to figure out all the technical parts if they were motivated by the ideas. It might just take them a bit longer to get the technical setup going. This is where I'm hoping to balance out the mentees team with a mix of engineering/ technical talent + someone with a motivated researcher mindset and interest in moral alignment.

Finally, I am also happy to guide a broader project that fits within the interests I listed above, if there is a motivated mentee with their own theoretical and practical idea.

• comfortable & highly proficient using Python
• experience training ML models in Python (e.g. PyTorch; HuggingFace)
• solid understanding of LLM post-training techniques
• research interest in alignment & social intelligence in AI

~ 5-10 hrs / week

fit to UK time zone