Foundational models with billions of parameters which have been trained on large corpora of data have demonstrated non-trivial skills in a variety of domains. However, due to their monolithic structure, it is challenging and expensive to augment them or impart new skills. On the other hand, due to their adaptation abilities, several new instances of these models are being trained towards new domains and tasks. In this work, we study the problem of efficient and practical composition of existing foundation models with more specific models to enable newer capabilities. To this end, we propose CALM -- Composition to Augment Language Models -- which introduces cross-attention between models to compose their representations and enable new capabilities. Salient features of CALM are: (i) Scales up LLMs on new tasks by 're-using' existing LLMs along with a few additional parameters and data, (ii) Existing model weights are kept intact, and hence preserves existing capabilities, and (iii) Applies to diverse domains and settings. We illustrate that augmenting PaLM2-S with a smaller model trained on low-resource languages results in an absolute improvement of up to 13\% on tasks like translation into English and arithmetic reasoning for low-resource languages. Similarly, when PaLM2-S is augmented with a code-specific model, we see a relative improvement of 40\% over the base model for code generation and explanation tasks -- on-par with fully fine-tuned counterparts.

翻译：具有数十亿参数并在大规模语料上训练的基础模型已在多个领域展现出显著能力。然而，由于这类模型的单体结构，对其进行增强或赋予新技能既困难又昂贵。另一方面，凭借其适应能力，这些模型的新实例正不断被训练以应对新领域和新任务。本文研究如何通过高效实用的方式将现有基础模型与更专门的模型进行组合，从而催生新能力。为此，我们提出CALM——组合式语言模型增强方法——该方法通过引入模型间的交叉注意力机制来组合其表征，进而实现新能力。CALM的显著特征包括：(i) 通过"复用"现有大语言模型，仅需少量额外参数与数据即可在新任务上扩展大语言模型规模；(ii) 保持现有模型权重不变，从而保留原有能力；(iii) 适用于多样化的领域与场景。实验表明，将PaLM2-S与针对低资源语言训练的较小模型组合后，在英语翻译和低资源语言数学推理等任务上可实现高达13%的绝对性能提升。类似地，当PaLM2-S与专用代码模型组合时，代码生成与解释任务相对基础模型提升40%，与完全微调的模型性能相当。