Large language models have achieved impressive performance on various natural language processing tasks. However, so far they have been evaluated primarily on benchmarks where all information in the input context is relevant for solving the task. In this work, we investigate the distractibility of large language models, i.e., how the model problem-solving accuracy can be influenced by irrelevant context. In particular, we introduce Grade-School Math with Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant information in the problem description. We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included. We also identify several approaches for mitigating this deficiency, such as decoding with self-consistency and adding to the prompt an instruction that tells the language model to ignore the irrelevant information.

翻译：大型语言模型在各种自然语言处理任务上已取得显著性能。然而，迄今为止，对其评估主要基于输入上下文中的所有信息均与任务求解相关的基准测试。本研究探讨了大型语言模型的可干扰性，即不相关上下文如何影响模型的问题求解准确率。具体而言，我们引入了包含不相关信息的算术推理数据集——带不相关上下文的小学数学题（GSM-IC），并利用该基准测试衡量前沿提示技术对大型语言模型的可干扰性。研究发现，当包含不相关信息时，模型性能显著下降。此外，我们还识别出几种缓解此缺陷的方法，例如采用自一致性解码，以及在提示中添加指令，要求语言模型忽略不相关信息。