Large language models (LLMs) can learn from a few demonstrations provided at inference time. We study this in-context learning phenomenon through the lens of Gaussian Processes (GPs). We build controlled experiments where models observe sequences of multivariate scalar-valued function samples drawn from known GP priors. We evaluate prediction error in relation to the number of demonstrations and compare against two principled references: (i) an empirical GP-regression learner that gives a lower bound on achievable error, and (ii) the expected error of a 1-nearest-neighbor (1-NN) rule, which gives a data-driven upper bound. Across model sizes, we find that LLM learning curves are strongly influenced by the function-generating kernels and approach the GP lower bound as the number of demonstrations increases. We then study the inductive biases of these models using a likelihood-based analysis. We find that LLM predictions are most likely under less smooth GP kernels. Finally, we explore whether post-training can shift these inductive biases and improve sample-efficiency on functions sampled from GPs with smoother kernels. We find that both reinforcement learning and supervised fine-tuning can effectively shift inductive biases in the direction of the training data. Together, our framework quantifies the extent to which LLMs behave like GP learners and provides tools for steering their inductive biases for continuous function learning tasks.

翻译：大语言模型（LLMs）能够在推理时通过少量演示样本进行学习。我们通过高斯过程（GPs）的视角研究这种上下文学习现象。我们设计了受控实验，让模型观测从已知GP先验中抽取的多元标量值函数样本序列。我们评估了预测误差与演示样本数量之间的关系，并与两个理论参考基准进行了比较：（i）一个经验GP回归学习器，它给出了可达到误差的下界；（ii）1-最近邻（1-NN）规则的期望误差，它提供了一个数据驱动的误差上界。在不同模型规模下，我们发现LLM的学习曲线显著受函数生成核的影响，并且随着演示样本数量的增加逐渐接近GP下界。随后，我们基于似然的分析方法研究了这些模型的归纳偏好。我们发现LLM的预测在平滑性较低的GP核下具有最高似然度。最后，我们探讨了后训练是否能够改变这些归纳偏好，并提升在平滑核GP采样函数上的样本效率。我们发现，无论是强化学习还是监督微调，都能有效地使归纳偏好向训练数据的方向调整。综合而言，我们的框架量化了LLMs在多大程度上表现出类似GP学习器的行为，并为引导其在连续函数学习任务中的归纳偏好提供了工具。