Fully-parametric language models generally require a huge number of model parameters to store the necessary knowledge for solving multiple natural language tasks in zero/few-shot settings. In addition, it is hard to adapt to the evolving world knowledge without the costly model re-training. In this paper, we develop a novel semi-parametric language model architecture, Knowledge-in-Context (KiC), which empowers a parametric text-to-text language model with a knowledge-rich external memory. Specifically, the external memory contains six different types of knowledge: entity, dictionary, commonsense, event, script, and causality knowledge. For each input instance, the KiC model adaptively selects a knowledge type and retrieves the most helpful pieces of knowledge. The input instance along with its knowledge augmentation is fed into a text-to-text model (e.g., T5) to generate the output answer, where both the input and the output are in natural language forms after prompting. Interestingly, we find that KiC can be identified as a special mixture-of-experts (MoE) model, where the knowledge selector plays the role of a router that is used to determine the sequence-to-expert assignment in MoE. This key observation inspires us to develop a novel algorithm for training KiC with an instance-adaptive knowledge selector. As a knowledge-rich semi-parametric language model, KiC only needs a much smaller parametric part to achieve superior zero-shot performance on unseen tasks. By evaluating on 40+ different tasks, we show that KiC_Large with 770M parameters easily outperforms large language models (LMs) that are 4-39x larger by a large margin. We also demonstrate that KiC exhibits emergent abilities at a much smaller model scale compared to the fully-parametric models.

翻译：全参数化语言模型通常需要海量模型参数来存储必要知识，以在零样本/少样本场景中解决多种自然语言任务。此外，这类模型难以适应不断演化的世界知识，且需付出高昂的模型重训练代价。本文提出了一种新型半参数化语言模型架构——知识语境（Knowledge-in-Context, KiC），该架构通过知识丰富的外部存储器增强参数化文本到文本语言模型的能力。具体而言，外部存储器包含六种知识类型：实体知识、词典知识、常识知识、事件知识、脚本知识与因果知识。针对每个输入实例，KiC模型自适应选择知识类型并检索最具助益的知识片段。输入实例与其知识增强结果共同输入文本到文本模型（如T5），通过提示（prompting）后以自然语言形式生成输出答案。有趣的是，我们发现KiC可被识别为一种特殊的混合专家（Mixture-of-Experts, MoE）模型，其中知识选择器起到路由器作用，用于确定MoE中序列到专家的分配关系。这一关键发现启发我们开发了新型算法，通过实例自适应的知识选择器训练KiC。作为知识丰富的半参数化语言模型，KiC仅需较小的参数化部分即可在未见任务上实现卓越的零样本性能。通过在40余种不同任务上的评估，我们展示了770M参数的KiC_Large模型以显著优势轻松超越体量4-39倍的更大规模语言模型。同时证明，与全参数化模型相比，KiC在更小的模型规模下即展现出涌现能力。