Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts, but there has been limited understanding of exactly how these explanations function or why they are effective. This work aims to better understand the mechanisms by which explanations are used for in-context learning. We first study the impact of two different factors on the performance of prompts with explanations: the computation trace (the way the solution is decomposed) and the natural language used to express the prompt. By perturbing explanations on three controlled tasks, we show that both factors contribute to the effectiveness of explanations. We further study how to form maximally effective sets of explanations for solving a given test query. We find that LLMs can benefit from the complementarity of the explanation set: diverse reasoning skills shown by different exemplars can lead to better performance. Therefore, we propose a maximal marginal relevance-based exemplar selection approach for constructing exemplar sets that are both relevant as well as complementary, which successfully improves the in-context learning performance across three real-world tasks on multiple LLMs.

翻译：大语言模型（LLMs）在从提示中的解释进行学习方面展现出卓越能力，但人们对其解释的具体作用机制及有效性成因仍知之甚少。本研究旨在深入理解解释在上下文学习中的应用机理。首先，我们系统探究了两个关键因素对含解释提示性能的影响：计算轨迹（问题求解的分解方式）与表达提示的自然语言。通过在三个受控任务中对解释进行扰动实验，证明这两个因素均对解释效能具有贡献作用。进一步地，我们研究如何构建针对特定测试查询的最优解释集。研究发现，大语言模型能从解释集的互补性中获益：不同示例展现的多样化推理能力可显著提升模型表现。基于此，我们提出一种基于最大边际相关性的示例选择方法，用于构建兼具相关性与互补性的示例集。该方法在多个大语言模型的三项实际任务中成功提升了上下文学习性能。