In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from multiple information corpora ("external memories"), with the option to "plug in" new memory at inference time. We develop a joint learning mechanism that trains the augmentation component with latent labels derived from the end retrieval task, paired with hard negatives from the memory mixture. We instantiate the model in a zero-shot dense retrieval setting by augmenting a strong T5-based retriever with MoMA. Our model, MoMA, obtains strong zero-shot retrieval accuracy on the eighteen tasks included in the standard BEIR benchmark. It outperforms systems that seek generalization from increased model parameters and computation steps. Our analysis further illustrates the necessity of augmenting with mixture-of-memory for robust generalization, the benefits of augmentation learning, and how MoMA utilizes the plug-in memory at inference time without changing its parameters. We plan to open source our code.

翻译：本文通过混合记忆增强（MoMA）机制提升语言模型的零样本泛化能力。该机制从多个信息语料库（"外部记忆"）中检索增强文档，并支持在推理阶段"插入"新记忆。我们提出一种联合学习机制，利用端到端检索任务生成的隐式标签，结合混合记忆中的难负样本训练增强组件。通过将强T5基础检索器与MoMA相结合，我们在零样本密集检索场景中实例化该模型。模型MoMA在标准BEIR基准测试包含的18项任务上展现出强劲的零样本检索准确率，其性能超越通过增加模型参数和计算步骤寻求泛化的系统。进一步分析揭示了：混合记忆增强对鲁棒泛化的必要性、增强学习的优势，以及MoMA在无需修改参数的情况下在推理阶段利用可插拔记忆的机制。我们计划开源相关代码。