Large Language Models (LLMs) have demonstrated remarkable capabilities in performing complex tasks. Moreover, recent research has shown that incorporating human-annotated rationales (e.g., Chain-of-Thought prompting) during in-context learning can significantly enhance the performance of these models, particularly on tasks that require reasoning capabilities. However, incorporating such rationales poses challenges in terms of scalability as this requires a high degree of human involvement. In this work, we present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY), which addresses the aforementioned challenges by automating the process of rationale generation. To this end, we leverage post hoc explanation methods which output attribution scores (explanations) capturing the influence of each of the input features on model predictions. More specifically, we construct automated natural language rationales that embed insights from post hoc explanations to provide corrective signals to LLMs. Extensive experimentation with real-world datasets demonstrates that our framework, AMPLIFY, leads to prediction accuracy improvements of about 10-25% over a wide range of tasks, including those where prior approaches which rely on human-annotated rationales such as Chain-of-Thought prompting fall short. Our work makes one of the first attempts at highlighting the potential of post hoc explanations as valuable tools for enhancing the effectiveness of LLMs. Furthermore, we conduct additional empirical analyses and ablation studies to demonstrate the impact of each of the components of AMPLIFY, which, in turn, leads to critical insights for refining in-context learning.

翻译：大型语言模型在复杂任务中展现出卓越能力。近期研究表明，在上下文学习中引入人工标注的推理链（如思维链提示）可显著提升模型性能，尤其在需要推理能力的任务中。然而，这种方法因需要大量人工参与而面临可扩展性挑战。本研究提出新型框架AMPLIFY（通过事后解释的上下文学习增强模型性能），通过自动化推理链生成流程解决上述挑战。具体而言，我们利用事后解释方法输出归因分数（解释结果），捕捉各输入特征对模型预测的影响。更关键的是，我们构建了嵌入事后解释洞见的自动化自然语言推理链，为语言模型提供修正信号。基于真实数据集的广泛实验表明，AMPLIFY框架在多项任务中实现约10-25%的预测准确率提升，甚至优于依赖人工标注推理链（如思维链提示）的现有方法。本研究首次系统论证了事后解释作为提升大语言模型效能工具的巨大潜力。此外，我们通过额外的实证分析和消融实验，揭示了AMPLIFY各组件的贡献，为优化上下文学习提供了关键洞见。