In-context learning is a key paradigm in large language models (LLMs) that enables them to generalize to new tasks and domains by simply prompting these models with a few exemplars without explicit parameter updates. Many attempts have been made to understand in-context learning in LLMs as a function of model scale, pretraining data, and other factors. In this work, we propose a new mechanism to probe and understand in-context learning from the lens of decision boundaries for in-context binary classification. Decision boundaries are straightforward to visualize and provide important information about the qualitative behavior of the inductive biases of standard classifiers. To our surprise, we find that the decision boundaries learned by current LLMs in simple binary classification tasks are often irregular and non-smooth, regardless of linear separability in the underlying task. This paper investigates the factors influencing these decision boundaries and explores methods to enhance their generalizability. We assess various approaches, including training-free and fine-tuning methods for LLMs, the impact of model architecture, and the effectiveness of active prompting techniques for smoothing decision boundaries in a data-efficient manner. Our findings provide a deeper understanding of in-context learning dynamics and offer practical improvements for enhancing robustness and generalizability of in-context learning.

翻译：上下文学习是大语言模型（LLMs）中的关键范式，它使模型能够通过仅提供少量示例提示（无需显式参数更新）来泛化到新任务和领域。已有许多尝试从模型规模、预训练数据等因素的角度来理解LLMs中的上下文学习。在本工作中，我们提出一种新机制，从上下文二分类任务的决策边界视角来探究和理解上下文学习。决策边界易于可视化，并能提供关于标准分类器归纳偏差定性行为的重要信息。令人惊讶的是，我们发现当前LLMs在简单二分类任务中学到的决策边界常常是不规则且不平滑的，无论底层任务是否线性可分。本文研究了影响这些决策边界的因素，并探索了增强其泛化能力的方法。我们评估了多种途径，包括LLMs的无训练方法和微调方法、模型架构的影响，以及以数据高效方式平滑决策边界的主动提示技术的有效性。我们的研究结果提供了对上下文学习动态的更深入理解，并为增强上下文学习的鲁棒性和泛化能力提供了实用的改进方案。