Large Language Models (LLMs) have shown impressive performance in complex reasoning tasks through Chain-of-Thought (CoT) reasoning, allowing models to break down problems into manageable sub-tasks. However, existing CoT evaluation techniques either require annotated CoT data or fall short in accurately assessing intermediate reasoning steps, leading to high rates of false positives. In this paper, we formalize CoT reasoning in LLMs through an information-theoretic lens. Specifically, our framework quantifies the `information gain' at each reasoning step, enabling the identification of failure modes in LLMs without the need for expensive annotated datasets. We demonstrate the efficacy of our approach through extensive experiments on toy and GSM-8K data, where it significantly outperforms existing outcome-based methods by providing more accurate insights into model performance on individual tasks.

翻译：大语言模型（LLMs）通过思维链（CoT）推理在复杂推理任务中展现出卓越性能，该机制使模型能够将问题分解为可处理的子任务。然而，现有的CoT评估方法要么需要标注的CoT数据，要么难以准确评估中间推理步骤，导致较高的误判率。本文通过信息论视角对LLMs中的CoT推理进行形式化建模。具体而言，我们提出的框架量化每个推理步骤的“信息增益”，从而无需依赖昂贵的标注数据集即可识别LLMs的失效模式。通过在玩具数据集和GSM-8K数据集上的大量实验，我们验证了该方法的有效性：相较于现有基于结果评估的方法，本框架能针对具体任务提供更精确的模型性能分析，显著提升了评估效能。