The emergent ability of Large Language Models to use a small number of examples to learn to perform in novel domains and tasks, also called in-context learning (ICL). In this work, we show that a much smaller model can be trained to perform ICL by fine-tuning towards a specialized training objective, exemplified on the task of domain adaptation for neural machine translation. With this capacity for ICL, the model can take advantage of relevant few-shot examples to adapt its output towards the domain. We compare the quality of this domain adaptation to traditional supervised techniques and ICL with a 40B-parameter Large Language Model. Our approach allows efficient batch inference on a mix of domains and outperforms state-of-the-art baselines in terms of both translation quality and immediate adaptation rate, i.e. the ability to reproduce a specific term after being shown a single example.

翻译：大语言模型（LLMs）展现出一种涌现能力：能够利用少量示例在陌生领域和任务中学习执行，这种能力也被称为上下文学习（ICL）。在本研究中，我们展示了一个规模小得多的模型可以通过针对特定训练目标进行微调来学习执行ICL，并以神经机器翻译的领域自适应任务为例进行验证。凭借这种ICL能力，模型能利用相关的少样本示例调整其输出以适应目标领域。我们将这种领域自适应的质量与传统的监督学习技术以及具有400亿参数的大语言模型的ICL进行了比较。我们的方法支持对混合领域进行高效的批量推理，在翻译质量和即时适应率（即展示单个示例后重现特定术语的能力）方面均超越了当前最先进的基线方法。