We want to isolate the outputs that LLMs produce that are not captured by traditional evaluation methods. We will do this by generating from the model without prompts, or allowing the model to generate its own evaluations.
About the project
What outputs do models produce that would never be discovered by traditional evaluations?
Evaluations like SimpleQA and SWE-Bench test models on a very well-understood area of their output distributions. But models are much more than this. We want to develop the ability to take a model (weights only) and infer its output distribution. Because this isn't conditional on any other evaluation data, we call this "data-free evaluation".
It seems likely that models are pretrained on a very wide distribution, aligned on a narrow distribution, and tested on an extremely narrow distribution. But there are lots of topics that e.g. Claude would be happy to talk about but has never been asked about before.
This project aims to uncover interesting model outputs that are likely (under the model output distribution) but typically unseen by humans or automatic evaluations. We will build systems for generating outputs from models and "steering them" away from typical evaluation distributions. This will require new conditional generation approaches that can elicit likely-but-different model generations. Success will also condition on our engineering skill, as we need to build this kind of system from scratch.
References:
- Data-free evaluation is very related to the "trawling" written up in the Gemini Safety Report: https://storage.googleapis.com/deepmind-media/gemini/gemini_3_pro_fsf_report.pdf
- I worked on a related project recently about recovering data from language model weights: https://arxiv.org/abs/2506.15553
- Also relevant is a recent Google paper on extracting alignment from open models: https://arxiv.org/abs/2510.18554
- Another example of something that could be extracted from a sophisticated data-free evaluation system is Claude's "soul document", found in the weights of Opus 4.5: https://www.lesswrong.com/posts/vpNG99GhbBoLov9og/claude-4-5-opus-soul-document
Theory of change
We currently don't have a good way to "map" the full output distribution of LLMs. We only test them on benchmarks and report scores and interesting outputs found. This feels like a huge miss for safety; we should be able to analyze the full breadth of possible outputs.
Your role
- Engineering systems for data-free evaluation
- Designing experiments to show why data-free evaluation is more useful than other evaluation metrics along certain axes
- Running experiments to demonstrate qualitative and quantitative benefits of DFE
Prerequisites
- Needs to have basic experience training models in PyTorch
- Want someone who's a great software engineer in general (but not ML-specific)
- Preferably a person who's highly empirical and interested in running lots of experiments to build a simple mental model of what's going on
- The ability to go above and beyond, i.e. come up with and implement new ideas independently without necessarily waiting for a meeting
Time commitment
10 hours per week seems like a reasonable minimum.
Location preference
US time zone preferred; I'll be on PST.
Application question(s)
- What do you think you could learn about the training data of GPT-2 from the model itself – without knowing anything else about the model's training process?
- Why does alignment make data-free evaluation difficult?
- What are some ideas for model outputs that might surface through data-free evaluation but not through traditional evaluation?
- What is one concrete technical idea for generating model outputs that have high likelihood but are not covered by evaluation?
About the mentor
Jack received his PhD from Cornell University and is most well-known for his research on model capacity and representation inversion. His research showed that model representations can be losslessly invertible back to text and that GPT-style LLMs can memorize up to 3.6 bits-per-parameter. More recently he went viral online for uncovering the 'base model' hidden underneath the weights of the aligned model GPT-OSS model weights.