Two lines of work are taking the central stage in AI research. On the one hand, the community is making increasing efforts to build models that discard spurious correlations and generalize better in novel test environments. Unfortunately, the bitter lesson so far is that no proposal convincingly outperforms a simple empirical risk minimization baseline. On the other hand, large language models (LLMs) have erupted as algorithms able to learn in-context, generalizing on-the-fly to the eclectic contextual circumstances that users enforce by means of prompting. In this paper, we argue that context $\approx$ environment, and posit that in-context learning holds the key to better domain generalization. Via extensive theory and experiments, we show that paying attention to context$\unicode{x2013}\unicode{x2013}$unlabeled examples as they arrive$\unicode{x2013}\unicode{x2013}$allows our proposed In-Context Risk Minimization (ICRM) algorithm to zoom-in on the test environment risk minimizer, leading to significant out-of-distribution performance improvements. From all of this, two messages are worth taking home. Researchers in domain generalization should consider environment as context, and harness the adaptive power of in-context learning. Researchers in LLMs should consider context as environment to better structure data towards generalization.

翻译：两条研究路线正占据人工智能研究的中心舞台。一方面，学界正加大力度构建能摒弃虚假相关性、在新型测试环境中实现更好泛化的模型。遗憾的是，迄今为止的惨痛教训是，尚无任何方案能令人信服地超越简单的经验风险最小化基线。另一方面，大型语言模型作为能够进行上下文学习、通过即时提示动态适应用户强加的多变情境的算法而迅速崛起。本文认为，上下文 $\approx$ 环境，并主张上下文学习是实现更好领域泛化的关键。通过广泛的理论与实验，我们证明：关注上下文——即随到随学的无标注样本——能使我们提出的上下文风险最小化算法聚焦于测试环境的风险最小化器，从而显著提升分布外性能。综上所述，两条信息值得铭记。领域泛化研究者应将环境视为上下文，并善用上下文学习的自适应能力；而大型语言模型研究者则应将上下文视为环境，以更优方式组织数据以促进泛化。