Language models (LMs) can generate hallucinations and incoherent outputs, which highlights their weak context dependency. Cache-LMs, which augment LMs with a memory of recent history, can increase context dependency and have shown remarkable performance in diverse language generation tasks. However, we find that even with training, the performance gain stemming from the cache component of current cache-LMs is suboptimal due to the misalignment between the current hidden states and those stored in the memory. In this work, we present HistAlign, a new training approach to ensure good cache alignment such that the model receives useful signals from the history. We first prove our concept on a simple and synthetic task where the memory is essential for correct predictions, and we show that the cache component of HistAlign is better aligned and improves overall performance. Next, we evaluate HistAlign on diverse downstream language generation tasks, including prompt continuation, abstractive summarization, and data-to-text. We demonstrate that HistAlign improves text coherence and faithfulness in open-ended and conditional generation settings respectively. HistAlign is also generalizable across different model families, showcasing its strength in improving context dependency of LMs in diverse scenarios. Our code is publicly available at https://github.com/meetdavidwan/histalign

翻译：语言模型（LMs）可能会生成幻觉性和不连贯的输出，这凸显了其上下文依赖性的薄弱。缓存语言模型通过为LMs增加最近历史的记忆来增强上下文依赖性，并在多种语言生成任务中展现出卓越性能。然而，我们发现即使经过训练，当前缓存LM中缓存组件带来的性能提升仍不理想，原因在于当前隐藏状态与记忆中存储的状态之间存在对齐偏差。本文提出HistAlign——一种确保缓存良好对齐的新训练方法，使模型能从历史信息中获取有效信号。我们首先在需要记忆才能正确预测的简单合成任务上验证了概念，表明HistAlign的缓存组件实现了更优对齐并提升了整体性能。随后，我们在包括提示续写、抽象式摘要和数据到文本生成等多种下游语言生成任务上评估了HistAlign。我们证明了HistAlign分别在开放式和条件式生成场景中提升了文本连贯性与忠实度。HistAlign在不同模型家族间具有泛化能力，展现了其在多种场景下增强LMs上下文依赖性的优势。我们的代码已开源在https://github.com/meetdavidwan/histalign