We propose a method to teach multiple large language models (LLM) to collaborate by interleaving their generations at the token level. We model the decision of which LLM generates the next token as a latent variable. By optimizing the marginal likelihood of a training set under our latent variable model, the base LLM automatically learns when to generate itself and when to call on one of the ``assistant'' language models to generate, all without direct supervision. Token-level collaboration during decoding allows for a fusion of each model's expertise in a manner tailored to the specific task at hand. Our collaborative decoding is especially useful in cross-domain settings where a generalist base LLM learns to invoke domain expert models. On instruction-following, domain-specific QA, and reasoning tasks, we show that the performance of the joint system exceeds that of the individual models. Through qualitative analysis of the learned latent decisions, we show models trained with our method exhibit several interesting collaboration patterns, e.g., template-filling. Our code is available at https://github.com/clinicalml/co-llm.

翻译：我们提出了一种方法，通过以词元级交织生成的方式，教会多个大型语言模型进行协作。我们将决定由哪个语言模型生成下一个词元的过程建模为潜在变量。通过在这种潜在变量模型下优化训练集的边际似然，基础语言模型能够自动学习何时自行生成，何时调用“辅助”语言模型进行生成——这一切均无需直接监督。解码过程中的词元级协作使得每个模型的专业知识能够以针对特定任务定制的方式融合。这种协同解码在跨领域场景中尤其有用，例如通用型基础语言模型学会调用领域专家模型。在指令遵循、领域特定问答及推理任务中，我们展示了联合系统的性能超过了个体模型。通过对学习到的潜在决策进行定性分析，我们发现使用我们的方法训练的模型展现出了几种有趣的协作模式，例如模板填充。我们的代码开源在 https://github.com/clinicalml/co-llm。