Temporal Representational Geometry: We test whether the time horizon a language model is planning over can be read directly from its internal activations. Building on our finding that time horizons form a geometric manifold in single decisions, we extend it to multi-step planning, as a first step toward monitoring and steering how far ahead AI agents plan.
About the project
The project. Language models encode abstract concepts like time geometrically. In prior work (arXiv:2606.05194), we found that a model’s time horizon, from seconds to centuries, forms an ordinal manifold in its residual stream, and that activations at change-of-turn tokens carry both this geometry and the model’s temporal preference. That study covered single decisions.
The question. This project asks whether the same structure governs planning: as a model decomposes a task into steps across a multi-turn conversation, can the time horizon it is planning over be predicted from its activations? Why this matters for alignment is developed in the next section; in short, both myopic and scheming failures are failures of how far ahead a model plans, and a horizon we can read from activations is a horizon we can monitor.
The plan. We build a dataset generator and an activation-capture pipeline, validate it on small models, then run the largest dense Qwen (20–40B) that fits available hardware plus matched open-weight families, capturing residual-stream activations at boundary tokens and asking, via PCA, whether they order by horizon. The deliverable is a go/no-go signal on planning-time temporal geometry, enough to justify scaling toward horizon monitors and steering.
See more: https://docs.google.com/document/d/1PVTBUPojodhr3FRYN8e8MV6Uq8ffwG1ixtnUVk0OOjQ/edit?tab=t.0
Theory of change
Time horizon shapes decisions. Options are scored by temporal scope: a cost unacceptable over a week can be right over a decade. So time preference is fundamental to planning and to cooperation, where an agent accepts a present cost for a later shared payoff. The wrong horizon yields systematically wrong decisions even when values are right.
Alignment is partly temporal: it needs sound objectives and the right horizon. Two opposite failures:
- Myopia: over-weighting the immediate objective cuts corners. Coding agents saturate visible test suites while failing held-out tests, a gap that widens with task length (SpecBench, arxiv.org/abs/2605.21384). This is short-horizon reward-seeking that scales with horizon.
- Scheming: a long-horizon planner hides the plan, behaving under oversight while pursuing a misaligned goal for later. Frontier models already recognize deception as viable and act on it in agentic evals (Meinke et al. 2024, arxiv.org/abs/2412.04984).
Both concern how far ahead the model plans and whether that horizon is intended. The target is to detect and hold it while still tractable.
Activation geometry as a fail-safe. Post-training suffices for routine use; high-stakes deployment needs more. If the planning horizon is legible in activations, characterize its geometry, then monitor representations against that manifold at inference and correct drift. This treats interpretability as runtime infrastructure, not just diagnostic.
arXiv:2606.05194 established this for one-shot decisions; the path above needs it during planning. Hence a three-step chain, each step gating the next:
- Existence (this project). Does multi-turn planning carry stable temporal geometry at boundary tokens? Deliverable: the evidence.
- Monitors. If so, train probes reporting a plan's horizon per turn, cheaper than token-by-token chain-of-thought analysis.
- Control. If readable, steer drifting representations back to the intended horizon, a runtime lever needing no retraining.
Your role
Mentees will lead the day-to-day technical work. I'm flexible on the balance of autonomy and direction and will calibrate how much guidance I provide to each mentee's preference.
See more: https://docs.google.com/document/d/1PVTBUPojodhr3FRYN8e8MV6Uq8ffwG1ixtnUVk0OOjQ/edit?tab=t.0
Prerequisites
- Highly proficient in Python.
- Have run local LLM inference with PyTorch and HuggingFace transformers (loading checkpoints, chat templates, generation), beyond API-only usage.
- Have extracted and inspected internal activations from a transformer at least once, using hooks, nnsight, or TransformerLens (toy models and guided exercises such as ARENA count).
- Comfortable with the linear algebra behind PCA (projections, eigendecomposition) and with reading 2D/3D embedding plots.
- Able to commit at least 10 hours/week for the full duration and to work independently between meetings.
Application question(s)
- Suppose our first runs show no clean horizon structure at the change-of-turn tokens. Propose two concrete follow-up experiments that would distinguish “no signal,” “confounded signal,” and “signal at a different location (other tokens or layers),” and say what each outcome would imply.
- The observed geometry could reflect the literal horizon wording in the prompt rather than an internal planning representation. Describe one control experiment that separates these two explanations.
- Name one method other than PCA you would use to detect or quantify horizon structure in the activations, and say when it would be the better choice.
About the mentors
Independent AI Safety Researcher.
My interests encompass: • Interpretability, Functional Concepts and Personas: https://arxiv.org/abs/2606.05194 • Diversity and Homogenization in LLMs: https://arxiv.org/abs/2601.06116 • Evals + Bias Mitigation: https://www.unrulyabstractions.com/pdfs/differential_treatment.pdf
Research work presented at EurIPS, ICML, TAIS @ Oxford. Previously worked for 8+ years as a computer vision engineer in Augmented Reality and as a research engineer in Autonomous Vehicles.
More info: https://www.unrulyabstractions.com
I’m an independent AI safety researcher studying how feature manifolds encode and organize high-level concepts in language models. My current work focuses on localizing and analyzing representations of planning horizons in component activations.
Background in ML research, working at startups, facilitating BlueDot Impact's AGI Strategy course, and mentoring for AI Safety Camp