Attribution is a key concept in large language models (LLMs) as it enables control over information sources and enhances the factuality of LLMs. While existing approaches utilize open book question answering to improve attribution, factual datasets may reward language models to recall facts that they already know from their pretraining data, not attribution. In contrast, counterfactual open book QA datasets would further improve attribution because the answer could only be grounded in the given text. We propose Hallucination Augmented Recitations (HAR) for creating counterfactual datasets by utilizing hallucination in LLMs to improve attribution. For open book QA as a case study, we demonstrate that models finetuned with our counterfactual datasets improve text grounding, leading to better open book QA performance, with up to an 8.0% increase in F1 score. Our counterfactual dataset leads to significantly better performance than using humanannotated factual datasets, even with 4x smaller datasets and 4x smaller models. We observe that improvements are consistent across various model sizes and datasets, including multi-hop, biomedical, and adversarial QA datasets.

翻译：属性（Attribution）是大语言模型（LLMs）中的关键概念，因为它能够控制信息来源并增强LLMs的事实性。现有方法利用开放书问答（Open Book QA）来改善属性，但事实类数据集可能促使语言模型回忆预训练数据中已知的事实，而非真正的属性。相比之下，反事实开放书问答数据集能进一步改进属性，因为答案只能基于给定文本。我们提出幻觉增强引述（Hallucination Augmented Recitations, HAR），通过利用LLMs中的幻觉来创建反事实数据集以改进属性。以开放书问答为案例研究，我们证明使用反事实数据集微调的模型能够增强文本基础（text grounding），从而显著提升开放书问答性能，F1分数最高提升8.0%。与人工标注的事实数据集相比，即使使用4倍更小的数据集和4倍更小的模型，我们的反事实数据集也能带来显著更优的表现。我们观察到，这种改进在多种模型规模和数据集（包括多跳、生物医学和对抗性问答数据集）中保持一致。