While retrieval augmented generation (RAG) has been shown to enhance factuality of large language model (LLM) outputs, LLMs still suffer from hallucination, generating incorrect or irrelevant information. One common detection strategy involves prompting the LLM again to assess whether its response is grounded in the retrieved evidence, but this approach is costly. Alternatively, lightweight natural language inference (NLI) models for efficient grounding verification can be used at inference time. While existing pre-trained NLI models offer potential solutions, their performance remains subpar compared to larger models on realistic RAG inputs. RAG inputs are more complex than most datasets used for training NLI models and have characteristics specific to the underlying knowledge base, requiring adaptation of the NLI models to a specific target domain. Additionally, the lack of labeled instances in the target domain makes supervised domain adaptation, e.g., through fine-tuning, infeasible. To address these challenges, we introduce Automatic Generative Domain Adaptation (Auto-GDA). Our framework enables unsupervised domain adaptation through synthetic data generation. Unlike previous methods that rely on handcrafted filtering and augmentation strategies, Auto-GDA employs an iterative process to continuously improve the quality of generated samples using weak labels from less efficient teacher models and discrete optimization to select the most promising augmented samples. Experimental results demonstrate the effectiveness of our approach, with models fine-tuned on synthetic data using Auto-GDA often surpassing the performance of the teacher model and reaching the performance level of LLMs at 10 % of their computational cost.

翻译：尽管检索增强生成（RAG）已被证明能提升大语言模型（LLM）输出的事实准确性，LLM仍存在幻觉问题，即生成错误或无关信息。一种常见的检测策略是再次提示LLM，以评估其回答是否基于检索到的证据，但这种方法成本高昂。另一种方案是在推理时使用轻量级自然语言推理（NLI）模型进行高效的事实性验证。虽然现有的预训练NLI模型提供了潜在的解决方案，但在真实的RAG输入上，其性能仍逊色于大型模型。RAG输入比大多数用于训练NLI模型的数据集更为复杂，且具有底层知识库特有的属性，这要求NLI模型需针对特定目标领域进行自适应调整。此外，目标领域缺乏标注实例，使得有监督的领域自适应方法（例如微调）难以实施。为应对这些挑战，我们提出了自动生成式领域自适应（Auto-GDA）。该框架通过合成数据生成实现无监督领域自适应。与以往依赖人工设计过滤与增强策略的方法不同，Auto-GDA采用迭代过程，利用低效教师模型提供的弱标签持续提升生成样本的质量，并通过离散优化筛选最具潜力的增强样本。实验结果表明，使用Auto-GDA生成的合成数据进行微调的模型，其性能不仅常超越教师模型，更能以10%的计算成本达到LLM的性能水平。