Chain-of-Thought (CoT) explanations are widely used to interpret how language models solve complex problems, yet it remains unclear whether these step-by-step explanations reflect how the model actually reaches its answer, or merely post-hoc justifications. We propose Normalized Logit Difference Decay (NLDD), a metric that measures whether individual reasoning steps are faithful to the model's decision-making process. Our approach corrupts individual reasoning steps from the explanation and measures how much the model's confidence in its answer drops, to determine if a step is truly important. By standardizing these measurements, NLDD enables rigorous cross-model comparison across different architectures. Testing three model families across syntactic, logical, and arithmetic tasks, we discover a consistent Reasoning Horizon (k*) at 70--85% of chain length, beyond which reasoning tokens have little or negative effect on the final answer. We also find that models can encode correct internal representations while completely failing the task. These results show that accuracy alone does not reveal whether a model actually reasons through its chain. NLDD offers a way to measure when CoT matters.

翻译：思维链（CoT）解释被广泛用于理解语言模型如何解决复杂问题，但尚不清楚这种逐步解释是否反映了模型实际得出答案的方式，抑或仅仅是事后合理化。我们提出归一化对数几率差衰减（NLDD），这是一种衡量单个推理步骤是否忠实于模型决策过程的指标。我们的方法通过破坏解释中的单个推理步骤，并度量模型对其答案置信度的下降程度，来判断该步骤是否真正重要。通过标准化这些度量，NLDD能够实现跨不同架构的严格模型间比较。在句法、逻辑和算术任务上测试三个模型系列后，我们发现存在一个一致的推理视界（k*），位于链长度的70%至85%之间，超过该视界后，推理标记对最终答案几乎没有影响或产生负面影响。我们还发现，模型可能编码出正确的内部表示，但完全无法完成任务。这些结果表明，仅凭准确性无法揭示模型是否真正通过其链条进行推理。NLDD提供了一种衡量CoT何时具备有效性的方法。