We present a framework - Prompt, Generate, Train (PGT) - to efficiently develop a generative question-answering model for open-book question-answering over a proprietary collection of text documents. The framework adapts a retriever augmented generation model to the target domain using supervised finetuning and reinforcement learning with synthetic feedback in a few-shot setting. This yields an aligned, uncertainty calibrated model that is competitive with GPT-4 based in-context retrieval augmented generation in generating relevant answers at lower serving costs. The synthetic generation pipeline generates high quality synthetic training data musing a medium sized LLM, Flan-T5 XXL, and a novel consistency filtering scheme. The pipeline is designed to generate both abstractive and extractive questions that span the entire corpus. Using samples from this dataset, the framework fine-tunes a smaller RAG model comprising a dense retriever and a smaller sized LLM on samples from the dataset. In parallel, the framework trains a Reward model to score domain grounded answers higher than hallucinated answers. In the next phase, the framework aligns to the RAG model with the target domain using reinforcement learning. This step improves the RAG model's ability to generate grounded answers and ignore out of domain questions. In the final phase, the framework calibrates the model uncertainty for extractive question-answers. This is a desirable feature since the model can be integrated into a cascading system where the RAG model's answer is surfaced only when the model is confident of its answer.

翻译：摘要：本文提出一个框架——提示、生成、训练（PGT）——以高效构建面向专有文本语料集合的开放书本问答生成模型。该框架采用监督微调和基于合成反馈的强化学习，在少样本场景下将检索增强生成模型适配至目标领域，从而获得一个与基于GPT-4的上下文检索增强生成模型在生成相关答案方面具有竞争力、且服务成本更低的已对齐、不确定性校准模型。合成数据生成管道通过中型大语言模型Flan-T5 XXL及创新的共识过滤方案，生成高质量合成训练数据。该管道设计可同时生成覆盖全语料库的抽象型与抽取型问题。利用此数据集中的样本，框架对包含密集检索器与较小规模大语言模型的检索增强生成模型进行微调。同时，框架训练奖励模型以优先对基于领域的可靠答案进行评分，抑制幻觉式答案。下一阶段，框架通过强化学习将检索增强生成模型与目标领域对齐。此步骤提升模型生成可靠答案的能力，并过滤领域外问题。最终阶段，针对抽取型问答实现模型不确定性校准。这一特性具有重要价值：模型可集成至级联系统中，仅在其对答案高置信度时输出结果。