Chain-of-Thought (CoT) prompting can effectively elicit complex multi-step reasoning from Large Language Models~(LLMs). For example, by simply adding CoT instruction ``Let's think step-by-step'' to each input query of MultiArith dataset, GPT-3's accuracy can be improved from 17.7\% to 78.7\%. However, it is not clear whether CoT is still effective on more recent instruction finetuned (IFT) LLMs such as ChatGPT. Surprisingly, on ChatGPT, CoT is no longer effective for certain tasks such as arithmetic reasoning while still keeping effective on other reasoning tasks. Moreover, on the former tasks, ChatGPT usually achieves the best performance and can generate CoT even without being instructed to do so. Hence, it is plausible that ChatGPT has already been trained on these tasks with CoT and thus memorized the instruction so it implicitly follows such an instruction when applied to the same queries, even without CoT. Our analysis reflects a potential risk of overfitting/bias toward instructions introduced in IFT, which becomes more common in training LLMs. In addition, it indicates possible leakage of the pretraining recipe, e.g., one can verify whether a dataset and instruction were used in training ChatGPT. Our experiments report new baseline results of ChatGPT on a variety of reasoning tasks and shed novel insights into LLM's profiling, instruction memorization, and pretraining dataset leakage.

翻译：思维链（Chain-of-Thought, CoT）提示能有效激发大语言模型（LLMs）复杂多步推理能力。例如，仅需在MultiArith数据集的每个输入查询中添加CoT指令"让我们一步步思考"，GPT-3的准确率即可从17.7%提升至78.7%。然而，对于更近期的指令微调（IFT）LLMs（如ChatGPT），CoT是否仍然有效尚不明确。令人惊讶的是，在ChatGPT上，CoT对算术推理等特定任务不再有效，但对其他推理任务仍保持有效性。此外，在前述任务中，ChatGPT通常能达到最佳性能，且无需显式指示即可生成CoT。因此，ChatGPT很可能已在这些任务上经过CoT训练并记住了该指令，从而在应用于相同查询时（即使没有CoT）隐式遵循该指令。我们的分析揭示了IFT引入的指令可能带来过拟合/偏见风险，这一问题在LLMs训练中日益普遍。同时，这暗示了预训练配方可能存在泄漏——例如，可验证某数据集和指令是否被用于训练ChatGPT。实验报告了ChatGPT在多种推理任务上的新基准结果，并提出了关于LLM特性刻画、指令记忆化和预训练数据集泄漏的新见解。