Probe by Proxy

As models become more capable and deployment patterns mature, the number of agents in deployment and the independence they're given will grow rapidly. Currently, proprietary models dominate enterprise deployments. GDM and Anthropic both use probe-to-judge cascades in production as part of their misuse and jailbreak safeguards. But there are a number of reasons why centralized probes from the labs are unlikely to be effective in the specific context of a particular enterprise deployment. We have early results, including a replication of the original Apollo deception probe work, that proxy probes retain the large majority of the signal. That is, send the full closed weight rollout through an open weight model and detect activation signals there.

We want to better understand this mechanism. For what models is it effective? For what kinds of risks and failures (hallucination, deception, cover up, sycophancy, ...)? What can we learn about why it succeeds or fails?

A lot of the research motivating our work was inspired by Apollo's deception probe work (https://www.apolloresearch.ai/science/detecting-strategic-deception-using-linear-probes/). Some important more recent work: https://arxiv.org/abs/2601.11516, https://arxiv.org/abs/2502.03407, https://arxiv.org/abs/2601.04603.

The enterprise deployment surface is under-focused by the AI safety community. Currently LLM judges can usually do a good job detecting specific misbehavior. But as the AI control community has demonstrated, as models become more and more capable, relying on LLM judges becomes riskier. On top of that, just practically having judges double and triple check AI actions is expensive. Businesses will bias towards doing the minimum to reduce enterprise risk, with no investment in preventing broader risk.

One solution is leveraging activations to detect unwanted behavior. A probe is ~free relative to a generative forward pass. And unless a model is trained with probes as an active gatekeeper, it is unlikely to be able to avoid activation-based detection effectively. Probe-to-judge cascades are already in production at frontier labs to detect misaligned behavior.

But there are several reasons centralized probes delivered by the labs won’t always be effective in deployment. The definition of acceptable is not universal. Acceptable according to Anthropic’s central training might not be acceptable for a given business. Probes are also sensitive to distribution shift. Some failures only exist at the long trajectory or fleet level and cannot be seen from inside a single production context. Enterprise deployers have access to more contextual information than the frontier labs, things like IT environment, code base, industry, desired risk profile, business goals and preferences, vendor context and integration, partner context and integration, client context and integration, etc.

As deployments mature and become more widespread and agents are given more independence, the risk introduced on the deployment side will grow dramatically. The risk of things like distributed rogue deployments and exfiltration will explode. Scaled capabilities to provide activation based monitoring for deployed agents will become a critical piece of the safety landscape.

We will dedicate substantial time (2+ hrs/week) to collaboration and mentorship, helping design and guide the project. A mentee must be able to take the structure we agree on together and independently execute. Note independently here isn't a theological commitment to zero communication outside specified times. We just mean that a mentee is expected to be able to execute well without continuous supervision and guidance.

• Highly proficient programmer. Python experience preferred.
• Has spent time working with transformers. Mech int experience is a plus. Experience pre and/or post training transformers also works. Basically, thinking about weights and gradients and activations can't be new.
• Experimental experience and understanding. This might be most important. Do you know how to think through the structure of a dataset and a testing regime and understand the ways the results may or may not provide evidence for the question you're trying to answer? When results are positive, do you know how to design the next test to eliminate a possible confounder or unrelated cause? When results are negative, how do you design the next experiment to hone in on an explanation for what to fix?

We prefer not to work asynchronously. So mentees in the eastern hemishpere would need to be open to working at odd hours so there's decent overlap with US west coast working hours. Western hemisphere is easiest.