We propose LoRA-MCL, a training scheme that extends next-token prediction in language models with a method designed to decode diverse, plausible sentence continuations at inference time. Traditional language modeling is an intrinsically ill-posed problem: given a context, multiple ``futures'' may be equally plausible. Our approach leverages Multiple Choice Learning (MCL) and the Winner-Takes-All loss to efficiently handle ambiguity through Low-Rank Adaptation. We provide a theoretical interpretation of applying MCL to language modeling, assuming the data is generated from a mixture of distributions. We illustrate the proposed approach using mixtures of Markov chains. We then demonstrate with experiments on visual and audio captioning, as well as machine translation, that our method achieves high diversity and relevance in generated outputs. The accompanying code and a general-purpose package for applying LoRA-MCL to a wide range of language models are made available.

翻译：我们提出LoRA-MCL，一种扩展语言模型中下一词预测的训练方案，该方法旨在推理时解码多样且合理的句子延续。传统语言建模本质上是一个不适定问题：给定上下文，可能存在多个同样合理的“未来”可能。我们的方法利用多选学习（MCL）和赢家通吃损失，通过低秩适配（LoRA）高效处理歧义性。我们为MCL在语言建模中的应用提供了理论解释，假设数据由混合分布生成。我们通过马尔可夫链混合模型阐释了所提方法。随后在视觉与音频描述生成以及机器翻译任务上的实验表明，我们的方法在生成输出中实现了高多样性与相关性。我们同时公开了配套代码及一个通用工具包，可用于将LoRA-MCL应用于各类语言模型。