Interpretability-guided fine-tuning promises to tell us which layers to adapt, but our Spring 2026 results show feature geography alone can't yet deliver: base and instruction-tuned models with nearly identical SAE feature geography demand opposite layer choices. This project identifies the missing signal (adapter-update structure, training regime, Jacobian-lens readouts) and builds it into a reliable, mechanistically justified layer-selection procedure.
About the project
Can interpretability tell us where to fine-tune? A growing 2026 literature says yes, e.g., SAE-derived layer scores for task-vector injection ("SAEs as Stethoscopes, Not Scalpels," arXiv:2605.28649), activation-guided LoRA placement, and now Anthropic's Jacobian lens (J-lens, July 2026), a cheap, weight-grounded per-layer readout of the concepts a model is poised to produce, all point toward mechanistically informed adaptation.
Our Spring 2026 SPAR results identify the gap between that promise and current tools: base and instruction-tuned variants of Gemma-2 and Qwen show shape-identical SAE feature-divergence profiles, yet opposite LoRA layer-selection verdicts (late-layer LoRA Pareto-better for IT models; all-layer needed for base models, at the cost of catastrophic forgetting). Moreover, divergent-feature composition rotates with depth even when the depth profile is conserved. Diagnosis is not yet prescription. This project closes the gap.
Workstreams (each ownable by one mentee): (1) Two-instrument diagnosis — compute J-lens-based divergence profiles alongside SAE metrics across base/IT pairs; agreement and disagreement between the instruments is itself evidence about what each measures, and J-lens sidesteps the base-SAE-on-IT-activations confound since no SAE training is involved. (2) Adapter-structure analysis — per-layer weight profiles and subspace alignment of LoRA updates against both diagnostic profiles. (3) Training-regime axis — controlled SFT vs. RL (GRPO) comparison as the candidate explanation for the base/IT asymmetry. (4) Generalization — extend across families (Gemma, Qwen, Llama) and task types (MMLU, GSM8K, HumanEval, HellaSwag), with seeds and FVU-controlled comparisons, ending in a tested procedure: given a model and task, which layers to adapt, with mechanistic justification. Mentees inherit a running Spring codebase (LoRA grids, four completed SAE metrics variants, SAE Lens / Gemma Scope pipeline, shared compute) — open questions, not a blank page.
Target: workshop paper + arXiv preprint; authorship matched to contribution, clarified in week 1.
Theory of change
Fine-tuning is where deployed behavior is set and where safety training silently erodes. Specifically, we observe below-chance MMLU after indiscriminate LoRA on base models, and narrow fine-tuning is known to degrade safety behavior in unintended ways. A reliable, mechanistically grounded answer to "which layers should this adaptation touch, and why" enables minimal-footprint adaptation that leaves unrelated capabilities and safety training intact, and yields auditable artifacts relevant to fine-tuning oversight and provider review of customer fine-tunes. Boundary results also improve the field's calibration on what interpretability tools can certify.
Your role
Each mentee owns one workstream end-to-end (experiments, figures, corresponding paper section) with a brief written weekly update; I set direction, review results in detail, and handle integration and framing. Primary workstream ≈80% of time, optional secondary support role ≤20%. High autonomy within a workstream; scope changes agreed in meetings.
Prerequisites
- Highly proficient in Python and PyTorch.
- Fine-tuned a transformer LM (LoRA/PEFT counts; guided or toy projects fine). - Comfortable with HuggingFace Transformers and running GPU jobs independently (Colab/RunPod).
- Familiarity with mech interp probes like SAEs, SAE Lens, J-lens, etc.
- Comfort with linear algebra (subspaces, norms, rank) needed for the adapter-analysis workstream.
Location preference
No geographic restriction; mentees must be able to attend one of two weekly meeting slots anchored to Central European time (historically Saturday ~13:00 UTC and Wednesday ~14:00 UTC).
Application question(s)
- Base and instruction-tuned variants of a model show nearly identical SAE feature-divergence-by-layer profiles, yet respond oppositely to late-layer-only LoRA. Propose one hypothesis for why, and one experiment (≤$500 compute) that could falsify it. (300 words)
- The J-lens reads single-token concepts via an averaged Jacobian; SAEs decompose activations into learned features. Name one question about fine-tuning each instrument can answer that the other cannot. (150 words)
- Link to a technical writing sample (report, blog post, thesis chapter, or anything showing how you communicate results).
About the mentor
Yuxiao is an independent researcher in mechanistic interpretability. Before she was a postdoc at the Basque Center for Applied Mathematics (BCAM) and an AI Safety researcher at the Beneficial AI Foundation (BAIF). She was also a SERI MATS scholar in 2022 and a MATS scholar in 2023 both Summer and Winter tracks. Her research interests include information theory, probabilistic frameworks, and their applications for building more theoretically sound and trustworthy AI systems. She has a background in statistical inference, machine learning, and deep generative models. She completed her PhD in Electronic Engineering at Tsinghua University and has mentored research teams with SPAR and Algoverse.