In this work, we investigate how Large Language Models (LLMs) adapt their internal representations when encountering inputs of increasing difficulty, quantified as the degree of out-of-distribution (OOD) shift. We reveal a consistent and quantifiable phenomenon: as task difficulty increases, whether through harder reasoning questions, longer contexts, or adding answer choices, the last hidden states of LLMs become substantially sparser. In short, \textbf{\textit{the farther the shift, the sparser the representations}}. This sparsity--difficulty relation is observable across diverse models and domains, suggesting that language models respond to unfamiliar or complex inputs by concentrating computation into specialized subspaces in the last hidden state. Through a series of controlled analyses with a learning dynamic explanation, we demonstrate that this sparsity is not incidental but an adaptive mechanism for stabilizing reasoning under OOD. Leveraging this insight, we design \textit{Sparsity-Guided Curriculum In-Context Learning (SG-ICL)}, a strategy that explicitly uses representation sparsity to schedule few-shot demonstrations, leading to considerable performance enhancements. Our study provides new mechanistic insights into how LLMs internalize OOD challenges. The source code is available at the URL: https://github.com/MingyuJ666/sparsityLLM.

翻译：本文探究大语言模型在应对难度递增输入时，如何调整其内部表示，并以分布外偏移程度作为量化指标。我们揭示了一个一致且可量化的现象：随着任务难度增加——无论是通过更复杂的推理问题、更长的上下文，还是增加选项——大语言模型的最后一层隐藏状态显著变得稀疏。简言之，**偏移愈大，表示愈稀疏**。这种稀疏性与难度之间的关系在多种模型和领域中都可见，表明语言模型通过将计算集中于最后一层隐藏状态中的特殊子空间，来应对不熟悉或复杂的输入。通过一系列带有学习动态解释的受控分析，我们证明了这种稀疏性并非偶然，而是分布外条件下稳定推理的一种自适应机制。基于这一发现，我们设计了**稀疏性引导的课程上下文学习**策略，该策略显式利用表示稀疏性来安排少样本演示顺序，从而显著提升性能。本研究为大语言模型如何内化分布外挑战提供了新的机理见解。源代码可于https://github.com/MingyuJ666/sparsityLLM获取。