Chain-of-Thought (CoT) prompting can effectively elicit complex multi-step reasoning from Large Language Models~(LLMs). For example, by simply adding CoT instruction ``Let's think step-by-step'' to each input query of MultiArith dataset, GPT-3's accuracy can be improved from 17.7\% to 78.7\%. However, it is not clear whether CoT is still effective on more recent instruction finetuned (IFT) LLMs such as ChatGPT. Surprisingly, on ChatGPT, CoT is no longer effective for certain tasks such as arithmetic reasoning while still keeping effective on other reasoning tasks. Moreover, on the former tasks, ChatGPT usually achieves the best performance and can generate CoT even without being instructed to do so. Hence, it is plausible that ChatGPT has already been trained on these tasks with CoT and thus memorized the instruction so it implicitly follows such an instruction when applied to the same queries, even without CoT. Our analysis reflects a potential risk of overfitting/bias toward instructions introduced in IFT, which becomes more common in training LLMs. In addition, it indicates possible leakage of the pretraining recipe, e.g., one can verify whether a dataset and instruction were used in training ChatGPT. Our experiments report new baseline results of ChatGPT on a variety of reasoning tasks and shed novel insights into LLM's profiling, instruction memorization, and pretraining dataset leakage.

翻译：思维链（Chain-of-Thought, CoT）提示能够有效激发大型语言模型（LLMs）执行复杂的多步推理。例如，在MultiArith数据集的每个输入查询中简单添加CoT指令“让我们逐步思考”后，GPT-3的准确率从17.7%提升至78.7%。然而，对于更近期的指令微调（Instruction Finetuned, IFT）LLMs（如ChatGPT），CoT是否仍然有效尚不明确。令人意外的是，在ChatGPT上，CoT对某些任务（如算术推理）不再有效，但对其他推理任务仍保持效力。此外，在前一类任务中，ChatGPT通常能取得最佳性能，且即使在未受指令的情况下也能生成CoT。因此，ChatGPT可能已在包含CoT的这些任务上经过训练，从而记住了该指令，导致即使没有CoT提示，它在处理相同查询时也会隐含地遵循此类指令。我们的分析揭示了IFT引入的指令过拟合/偏差的潜在风险，这在LLMs训练中日益普遍。同时，这暗示了预训练配方的可能泄漏，例如，可以验证某个数据集和指令是否被用于训练ChatGPT。我们的实验报告了ChatGPT在多种推理任务上的新基线结果，并为了解LLMs的剖析、指令记忆化及预训练数据集泄漏提供了新颖见解。