Reasoning-trained language models can pick a document's few labels from thousands of candidates, zero-shot. We ask which components make these choices, and whether a student can be taught where to look as well as what to write, so its decisions trace to a few components and what they read. We study clinical coding of MIMIC-IV discharge summaries with all 5,651 candidate codes in the prompt. Each choice is one token: a region of the code list, a code within it, or the departure from a similar code named earlier. We measure each attention head's causal effect there by removing it and transplanting it into another summary. On essentially every summary, a small, shared set of heads, partly specific to each kind of choice, is necessary and sufficient; blocking what the tested heads read removes their effect. MISTILL builds on this: besides the teacher's text, it supervises where the student attends at the teacher's decision tokens. In a cross-family student, its own heads, found after training, restore nearly twice as much of the choice between similar codes as before, where chain-of-thought distillation alone adds under a third; one that already carried most of it gains a little, consistently across seeds, and both gains hold on the students' own completed traces. On two corpora with coder-marked evidence, a MISTILL student's decision heads attend to it more, and more specifically, than a chain-of-thought-distilled student's. Controls locate the gain in supervising attention at the decision tokens; in the cross-family student, a target built from unselected teacher heads loses it. No task difference from chain-of-thought distillation is detected at matched precision. The structure recurs in a second reasoner; ranked heads beat random ones on legal-document labelling and code generation; and without retraining the students keep their gain on Spanish case reports and a QA benchmark.
翻译:暂无翻译