An attempt to implement a reinforcement learning agent with infrabayesian rather than classical epistemology. The goal is a proof of concept RL agent which converges to optimal policies on Newcomb-like problems.
About the project
It is a classical theorem that ordinary reinforcement learning agents cannot converge to optimal policies on Newcomb's problem, as well as similar decision-theoretically complex problems, especially those with policy-dependent environments. Infrabayesianism, a recently developed epistemology which draws on both Bayesian epistemology and a form of imprecise probability along with a min rather than average for expected values, proposes to permit RL agents which can converge to such optimal policies, but existing work focuses entirely on regret bounds rather than working implementations. We will attempt to code up such an agent.
Theory of change
Newcomblike problems are a key aspect of alignment, and infrabayesianism has attracted substantial attention as a possible solution to cautious value-learning. A proof of concept of an RL agent that uses IB to solve Newcomb's problem (or similar decision-theory puzzles) would be a landmark result and confirmation of theory.
Your role
Mentees will be responsible for learning infrabayesianism in some depth and doing much of the necessary coding, development, and testing.
Prerequisites
Proficiency in Python, C++, or some other language suitable for coding toy reinforcement learning agents. A fairly deep mathematical background, ideally including plenty of probability theory and some real analysis.
Location preference
Preference for Pacific Time (UTC-8) and location in the Bay Area to permit in-person meetings, but my preference for this is not strong.
Application question(s)
Choose one, then do your own brief searching and reading to answer: (300 words)
- Explain Newcomb's problem. What makes it difficult? Why might an RL agent using classical probabilities have trouble converging to an optimal policy on it?
- Explain the Nirvana/Demiurge trick in infrabayesianism. What other foundational part of IB makes it work? Why do we need to use it?
- Come up with anything else novel and mathematically-grounded to say about the relationship between infrabayesianism and RL.
About the mentor
Paul is an agent foundations researcher who's turned from geometric group theory to questions of ontology, epistemology, causal modeling, honesty, and the substructure of agents. As of late 2025, he's now working with Francis Rhys Ward on an approach to the ELK problem.