In-context learning (ICL) has shown impressive results in few-shot learning tasks, yet its underlying mechanism is still not fully understood. Recent works suggest that ICL can be thought of as a gradient descent (GD) based optimization process. While promising, these results mainly focus on simplified settings of ICL and provide only a preliminary evaluation of the similarities between the two methods. In this work, we revisit the comparison between ICL and GD-based finetuning and study what properties of ICL an equivalent process must follow. We highlight a major difference in the flow of information between ICL and standard finetuning. Namely, ICL can only rely on information from lower layers at every point, while finetuning depends on loss gradients from deeper layers. We refer to this discrepancy as Layer Causality and show that a layer causal variant of the finetuning process aligns with ICL on par with vanilla finetuning and is even better in most cases across relevant metrics. To the best of our knowledge, this is the first work to discuss this discrepancy explicitly and suggest a solution that tackles this problem with minimal changes.

翻译：上下文学习（ICL）在小样本学习任务中展现出显著效果，但其内在机制尚未被完全理解。近期研究表明，ICL可被视为基于梯度下降（GD）的优化过程。尽管这一观点颇具前景，但这些研究主要关注ICL的简化场景，仅对两种方法的相似性进行了初步评估。本文重新审视了ICL与基于GD的微调之间的比较，并探究等价过程必须遵循的ICL属性。我们揭示了ICL与标准微调在信息流上的核心差异：ICL在每个阶段仅能依赖低层信息，而微调则依赖于深层损失梯度。我们将这一差异称为"层因果性"，并证明微调过程的层因果变体与ICL的对齐程度与普通微调相当，且在大部分相关指标上表现更优。据我们所知，这是首个明确讨论该差异并提出最小改动解决方案的研究。