Current language models are unable to quickly learn new concepts on the fly, often requiring a more involved finetuning process to learn robustly. Prompting in-context is not robust to context distractions, and often fails to confer much information about the new concepts. Classic methods for few-shot word learning in NLP, relying on global word vectors, are less applicable to large language models. In this paper, we introduce a novel approach named CoLLEGe (Concept Learning with Language Embedding Generation) to modernize few-shot concept learning. CoLLEGe is a meta-learning framework capable of generating flexible embeddings for new concepts using a small number of example sentences or definitions. Our primary meta-learning objective is simply to facilitate a language model to make next word predictions in forthcoming sentences, making it compatible with language model pretraining. We design a series of tasks to test new concept learning in challenging real-world scenarios, including new word acquisition, definition inference, and verbal reasoning, and demonstrate that our method succeeds in each setting without task-specific training.

翻译：当前语言模型无法即时快速学习新概念，通常需要更复杂的微调过程才能实现稳健学习。上下文提示对语境干扰缺乏鲁棒性，且往往难以有效传递新概念信息。自然语言处理中基于全局词向量的经典少样本词汇学习方法，在大型语言模型上已不再适用。本文提出名为CoLLEGe（概念学习与语言嵌入生成）的新方法，旨在革新少样本概念学习技术。CoLLEGe是一种元学习框架，能够通过少量示例句或定义生成新概念的灵活嵌入。我们的核心元学习目标仅在于帮助语言模型对后续句子进行下一词预测，这与语言模型预训练范式高度兼容。我们设计了一系列任务来测试真实场景下的新概念学习能力，包括新词习得、定义推理和语言推理，实验证明该方法无需特定任务训练即可在各场景中取得成功。