Emergent chain-of-thought (CoT) reasoning capabilities promise to improve performance and explainability of large language models (LLMs). However, uncertainties remain about how reasoning strategies formulated for previous model generations generalize to new model generations and different datasets. In this small-scale study, we compare different reasoning strategies induced by zero-shot prompting across six recently released LLMs (davinci-002, davinci-003, GPT-3.5-turbo, GPT-4, Flan-T5-xxl and Cohere command-xlarge) on a mixture of six question-answering datasets, including datasets from scientific and medical domains. Our findings demonstrate that while some variations in effectiveness occur, gains from CoT reasoning strategies remain robust across different models and datasets. GPT-4 has the most benefit from current state-of-the-art reasoning strategies and exhibits the best performance by applying a prompt previously discovered through automated discovery.

翻译：涌现的思维链推理能力有望提升大语言模型的表现与可解释性。然而，针对先前模型代次设计的推理策略如何泛化至新一代模型及不同数据集，仍存在不确定性。本研究通过小规模实验，在六种近期发布的大语言模型（davinci-002、davinci-003、GPT-3.5-turbo、GPT-4、Flan-T5-xxl 与 Cohere command-xlarge）上，采用零样本提示法，比较了由不同推理策略在六个问答数据集（含科学及医学领域数据集）混合场景下的表现。结果表明，尽管有效性存在部分差异，但思维链推理策略带来的增益在不同模型与数据集中保持稳健。GPT-4 从当前最优推理策略中获益最大，且通过应用先前经由自动发现方法获得的提示，展现出最佳性能。