Recent work has shown the promise of learning with human feedback paradigms to produce human-determined high-quality text. Existing works use human feedback to train large language models (LLMs) in general domain abstractive summarization and have obtained summary quality exceeding traditional likelihood training. In this paper, we focus on a less explored form of human feedback -- Human Edits. We propose Sequence Alignment (un)Likelihood Training (SALT), a novel technique to use both the human-edited and model-generated data together in the training loop. In addition, we demonstrate simulating Human Edits with ground truth summaries coming from existing training data -- Imitation edits, along with the model-generated summaries obtained after the training, to reduce the need for expensive human-edit data. In our experiments, we extend human feedback exploration from general domain summarization to medical domain summarization. Our results demonstrate the effectiveness of SALT to improve the summary quality with Human and Imitation Edits.

翻译：近期研究表明，利用人类反馈学习范式能够生成符合人类标准的高质量文本。现有工作通过人类反馈训练通用领域抽象摘要生成的大语言模型，获得的摘要质量已超越传统似然训练方法。本文聚焦于一种较少被探索的人类反馈形式——人类编辑。我们提出序列对齐（非）似然训练技术（SALT），这是一种在训练循环中同时利用人类编辑数据与模型生成数据的新方法。此外，我们证明可通过现有训练数据中的真实摘要模拟人类编辑（即模拟编辑），并结合训练后获得的模型生成摘要，减少对昂贵人工编辑数据的需求。在实验中，我们将人类反馈的探索领域从通用摘要扩展至医学摘要。实验结果验证了SALT利用人类编辑与模拟编辑提升摘要质量的有效性。