Build a pipeline for "emergent alignment". Can fine-tuning an open-source model (e.g. Gemma) on a single narrow good value, like compassion for animals, make it broadly more aligned toward humans too, without hurting its capabilities?
About the project
Update: we've expanded this project with 2 additional co-mentors (Constance & Aidan) and a new project!
Mentees under Allen will work on the emergent alignment project below - this will be a mix of fine-tuning models, developing evals, and other associated work, leading towards a published paper(s).
Mentees under Constance & Aidan will continue development of this reasoning pipeline, also working towards a published paper.
These are separate workstreams, but there will more than likely be shared insights, collaboration, and technical advisement across teams. If any questions on this new setup, please reach out to me at allen@projectmycelium.ai.
--
Recent work on "emergent misalignment" showed that fine-tuning a model on one narrow bad behavior (writing insecure code) makes it broadly misaligned — praising dictators, giving malicious advice. This project explores the inverse: if we train a model on one narrow good value, does it become broadly more aligned? If so, which value should that be?
The welfare of nonhuman animals seems like a strong candidate. It's a genuine moral blind spot that's incredibly unexplored in technical AI safety, despite AI systems increasingly making decisions that affect animals at scale — in agriculture, research, conservation, and beyond. There's also good reason to think the value would carry over and improve model alignment more generally.
We'll fine-tune an open-source model (Gemma, Qwen, etc.) on synthetic animal welfare datasets, then test how that model performs out of distribution (OOD). Does the model also become more compassionate toward humans, less biased against marginalized groups, and more robust on standard safety evals? How does this affect capabilities?
Early evidence suggests it can work. CaML's "Alignment midtraining for animals" found that animal-compassion training transferred to human compassion, and both Anthropic and Google DeepMind have recently shown that teaching a model the reasons behind good behavior generalizes far better than training on the behavior alone.
The project starts with an open-source data generation pipeline, modeled on Anthropic's "Teaching Claude Why" and DeepMind's follow-up experiments, producing two kinds of training data: synthetic documents and ethical dilemma Q&A, both grounded in a constitution for how AI should reason about sentient beings.
Core work: fine-tune Gemma on different combinations of this data (documents only, chat only, both), then evaluate on 1) animal welfare benchmarks, 2) benchmarks testing moral consideration for humans (safety, bias, discrimination), and 3) capability benchmarks (MMLU, HLE, etc.) to check for degradation.
Mentees will get hands-on experience with the full alignment workflow while working on an open question: whether a value instilled narrowly can generalize widely, and whether that points toward emergent alignment.
Relevant links: Alignment midtraining for animals: https://arxiv.org/abs/2604.13076 · Teaching Claude Why: https://alignment.anthropic.com/2026/teaching-claude-why/ · DeepMind Synthetic document finetuning for instilling positive traits: https://www.lesswrong.com/posts/GTYJRLhqztxKF2v5R/synthetic-document-finetuning-for-instilling-positive-traits · Emergent Misalignment (Betley et al.): https://arxiv.org/abs/2502.17424 · Existing animal welfare benchmark: https://www.mantabench.org/
Theory of change
As AI systems become more capable and deployed in high-stakes domains, they will increasingly make decisions affecting nonhuman welfare. If frontier models treat animal welfare as negligible, this creates two risks:
Direct harm scaling: AI systems optimizing for human preferences while discounting animal welfare could dramatically scale suffering through precision livestock farming, autonomous vehicles, and other animal-impacting technologies.
Value lock-in: As AI approaches AGI and superintelligence, current value frameworks risk becoming increasingly difficult to alter. Influencing model values now is critical, while systems are still relatively narrow.
This project addresses those risks by showing that greater moral consideration for animals in AI models can generalize, improving overall alignment. That gives labs like Anthropic and Google the evidence base to integrate nonhuman welfare into their AI frameworks (constitutions, model specs, etc.). Catching speciesist reasoning patterns early also lets us correct them before they scale with model capabilities.
Your role
Flexible, and happy to work in a way that works for you! Here are some options I see:
- Co-builders: Take ownership of major components and drive them to completion
- Contributors: Execute specific tasks like curriculum development, research, or content creation with guidance
- Supporters: Assist with tasks, such as research, logistics, and implementation
Prerequisites
Treat these as guides, not constraints! You definitely don’t need to have all of these requirements, and I’d encourage you to apply regardless. I’m very open to taking on mentees newer to the space, as I’m sure we’ll both learn a lot from each other.
Required:
- Background: Experience in topics around ethics, moral philosophy, computer science, and animal welfare (coursework, volunteer work, or demonstrated interest)
- Technical proficiency: knowledge of Python (other languages a plus), version control with Git, and can write clear technical documentation
- LLM APIs: Experience with integrating LLMs into projects or has built applications using LLM APIs (OpenAI, Claude API, etc.).
- AI-assisted coding: Regularly uses LLMs (ChatGPT, Claude, Cursor, etc.) as coding assistants and can effectively iterate on LLM-generated code.
- Research basics: Comfortable reading technical papers, identifying gaps in existing work, and synthesizing findings to iterate on previous research
- General: Strong communication skills, can communicate technical concepts clearly to non-technical audiences, self-starter mindset: works independently to make measurable progress between meetings
- Familiarity with eval topics (prompt engineering) or frameworks (e.g. Inspect) is a plus.
Application question(s)
*Word counts are suggestions, not constraints! Please feel free to go over if you have more you’d like to expand on, or under if not as much.
Question 1 (200 words):
- What draws you to work at the intersection of AI safety & animal welfare?
Describe your background, including (1) Relevant skills, coursework, or experience, and (2) How you've engaged with the field of AIxAnimals (i.e. read blog posts/articles/papers, done courses, fellowships, gone to events/meetups/conferences, etc.).
Question 2 (100 words):
- What’s your greatest accomplishment? Please provide a link to a relevant code sample, research paper, blog post, portfolio piece, etc. you've completed (does not necessarily have to be technical), along with a short description of what you did.
Question 3 (100 words):
- Review this existing eval. What’s one limitation or gap in how this benchmark handles questions about animals or non-human entities? Propose a specific test scenario or methodology improvement that would address this limitation. Some guiding questions: what would the prompt look like? What would you measure? How would you know if it's working?
About the mentors
Allen is the Founder and Executive Director of Mycelium, an organization working to make AI go well for all sentient beings through technical AI safety research & engineering. Currently, the org focuses on benchmarking, evals, and other alignment efforts through training data and SFT. Recent work involved developing MANTA, a multi-turn adversarial benchmark for measuring animal welfare values in frontier models: https://www.mantabench.org/
Concurrently, Allen is working with the NYU Center for Mind, Ethics, and Policy as a Technical Evals Researcher with Jeff Sebo and advisors from Google Deepmind. Previously, he led programming efforts at Collider, NYC's coworking hub for AI safety and other high-impact work. He has also facilitated the AGI strategy course for Bluedot, as well as AIxAnimals fellowships for Sentient Futures and Electric Sheep. He was also a mentor in a previous iteration of SPAR. Before transitioning into AI safety, he worked as a software engineer in creative technology.
Org website: https://www.projectmycelium.ai/
Aidan Kankyoku
Anima International
Aidan Kankyoku leads the Animal Welfare Alignment Team at Anima International, studying the impacts of different alignment methodologies on ethical behavior towards nonhuman minds and creating benchmarks and training data to teach models to reason responsibly about animal welfare. He publishes the Animal Welfare Alignment Newsletter on Substack, and separately writes about animal welfare strategy more broadly at Sandcastles. He has worn different hats as a campaigner and researcher in farmed animal welfare for ten years.
Constance Li
Sentient Futures
Constance Li is the founder and Executive Director of Sentient Futures, an organization focused future nonhuman welfare field-building. She also founded Hive, a digital community focused on farmed animal welfare. She was a co-author on AnimalHarmBench, the first benchmark for evaluating risks of animal harm in large language model-generated text. Her writing has appeared in Asterisk Magazine, and she's spoken on the Existential Hope podcast. In a past life, she was a physician specializing in physical medicine and rehab.