Large language models (LLMs) have showcased their capability with few-shot inference known as in-context learning. However, in-domain demonstrations are not always readily available in real scenarios, leading to cross-domain in-context learning. Besides, LLMs are still facing challenges in long-tail knowledge in unseen and unfamiliar domains. The above limitations demonstrate the necessity of Unsupervised Domain Adaptation (UDA). In this paper, we study the UDA problem under an in-context learning setting to adapt language models from the source domain to the target domain without any target labels. The core idea is to retrieve a subset of cross-domain elements that are the most similar to the query, and elicit language model to adapt in an in-context manner by learning both target domain distribution and the discriminative task signal simultaneously with the augmented cross-domain in-context examples. We devise different prompting and training strategies, accounting for different LM architectures to learn the target distribution via language modeling. With extensive experiments on Sentiment Analysis (SA) and Named Entity Recognition (NER) tasks, we thoroughly study the effectiveness of ICL for domain transfer and demonstrate significant improvements over baseline models.

翻译：大型语言模型（LLMs）已展现出通过上下文学习进行少样本推理的能力。然而，在实际场景中，领域内的示范样本并不总是容易获得，这导致了跨领域的上下文学习。此外，LLMs 在未知和不熟悉领域的长期知识方面仍面临挑战。上述局限性凸显了无监督领域自适应（UDA）的必要性。在本文中，我们研究了上下文学习设置下的 UDA 问题，旨在无需任何目标标签的情况下，将语言模型从源领域适应到目标领域。核心思想是检索与查询最相似的跨领域元素子集，并通过增强的跨领域上下文示例同时学习目标领域分布和判别任务信号，以上下文学习的方式引导语言模型进行适应。我们设计了不同的提示和训练策略，以适应不同的语言模型架构，并通过语言建模学习目标分布。通过在情感分析（SA）和命名实体识别（NER）任务上的大量实验，我们深入研究了上下文学习在领域迁移中的有效性，并展示了相对于基线模型的显著改进。