Compressing large language models reduces memory use and inference cost, but it can also create failures that standard benchmarks miss. A pruned model may still perform well on multiple-choice evaluations, yet fail to answer the same question in open generation. We ask what pruning changes: does it erase the correct answer, or does it make the answer harder to produce as the top output? We study this question with multilingual question answering, tracking the same questions before and after pruning. We find a benchmark illusion. Under high-sparsity pruning, especially Wanda, models often fail in greedy open generation while still selecting the correct answer under multiple-choice scoring. In these recognition-only errors, the answer is usually not gone, but demoted: it often reappears with beam search, sampling, or one in-context example. Overall, multiple-choice benchmarks can overstate the usability of compressed LLMs, creating an evaluation blind spot. Compressed models should be tested on what they can produce, not only on what they can recognize.

翻译：压缩大型语言模型可降低内存使用和推理成本，但也可能引发标准基准测试无法捕捉的故障。经剪枝的模型或仍在多项选择评估中表现良好，但在开放式生成任务中却无法回答相同问题。我们追问剪枝改变了什么：是抹去了正确答案，还是使答案更难成为最高概率输出？通过多语言问答任务，我们追踪同一问题在剪枝前后的表现，发现了一种基准测试幻觉。在高稀疏度剪枝（尤其是Wanda方法）下，模型在贪婪式开放生成中常失败，但仍能在多项选择评分下选出正确答案。在这些仅保留识别能力的错误中，答案通常未被删除而是被降级：通过束搜索、采样或单个上下文示例，答案往往能重新出现。总体而言，多项选择基准测试可能夸大压缩语言模型的可用性，形成评估盲区。压缩模型应测试其生成能力，而非仅验证其识别能力。