We propose LoRA-MCL, a training scheme that extends next-token prediction in language models with a method designed to decode diverse, plausible sentence continuations at inference time. Traditional language modeling is an intrinsically ill-posed problem: given a context, multiple futures may be equally plausible. Our approach leverages Multiple Choice Learning (MCL) and the winner-takes-all loss to efficiently handle ambiguity through Low-Rank Adaptation. We provide a theoretical interpretation of applying MCL to language modeling, assuming the data is generated from a mixture of distributions. We illustrate the proposed approach using mixtures of Markov chains. We then demonstrate with experiments on audio and visual captioning, as well as machine translation, that our method achieves high diversity and relevance in generated outputs. We release the code for applying LoRA-MCL to a wide range of language models.

翻译：我们提出LoRA-MCL，这是一种训练方案，通过一种旨在推理时解码多样且合理的句子续写的扩展方法，改进了语言模型中的下一个词预测。传统语言建模本质上是一个不适定问题：给定一个上下文，多个未来可能同样合理。我们的方法利用多选学习(MCL)和胜者全得损失，通过低秩适配高效处理歧义。我们对将MCL应用于语言建模提供了理论解释，假设数据来自混合分布。我们使用马尔可夫链混合来说明所提出的方法。随后通过音频和视觉字幕生成以及机器翻译的实验证明，我们的方法在生成输出中实现了高多样性和相关性。我们发布了将LoRA-MCL应用于各类语言模型的代码。