Recent research has sought to understand Transformers through the lens of in-context learning with functional data. We extend that line of work with the goal of moving closer to language models, considering categorical outcomes, nonlinear underlying models, and nonlinear attention. The contextual data are of the form $\textsf{C}=(x_1,c_1,\dots,x_N,c_{N})$ where each $c_i\in\{0,\dots,C-1\}$ is drawn from a categorical distribution that depends on covariates $x_i\in\mathbb{R}^d$. Contextual outcomes in the $m$th set of contextual data, $\textsf{C}_m$, are modeled in terms of latent function $f_m(x)\in\textsf{F}$, where $\textsf{F}$ is a functional class with $(C-1)$-dimensional vector output. The probability of observing class $c\in\{0,\dots,C-1\}$ is modeled in terms of the output components of $f_m(x)$ via the softmax. The Transformer parameters may be trained with $M$ contextual examples, $\{\textsf{C}_m\}_{m=1,M}$, and the trained model is then applied to new contextual data $\textsf{C}_{M+1}$ for new $f_{M+1}(x)\in\textsf{F}$. The goal is for the Transformer to constitute the probability of each category $c\in\{0,\dots,C-1\}$ for a new query $x_{N_{M+1}+1}$. We assume each component of $f_m(x)$ resides in a reproducing kernel Hilbert space (RKHS), specifying $\textsf{F}$. Analysis and an extensive set of experiments suggest that on its forward pass the Transformer (with attention defined by the RKHS kernel) implements a form of gradient descent of the underlying function, connected to the latent vector function associated with the softmax. We present what is believed to be the first real-world demonstration of this few-shot-learning methodology, using the ImageNet dataset.

翻译：近期研究试图通过函数数据上下文学习的视角来理解Transformer。为更贴近语言模型，本研究拓展了该方向的工作，考虑了分类结果、非线性基础模型以及非线性注意力机制。上下文数据的形式为$\textsf{C}=(x_1,c_1,\dots,x_N,c_{N})$，其中每个$c_i\in\{0,\dots,C-1\}$从依赖于协变量$x_i\in\mathbb{R}^d$的分类分布中抽取。第$m$组上下文数据$\textsf{C}_m$中的上下文结果通过隐函数$f_m(x)\in\textsf{F}$建模，其中$\textsf{F}$是具有$(C-1)$维向量输出的函数类。观测到类别$c\in\{0,\dots,C-1\}$的概率通过softmax函数依据$f_m(x)$的输出分量建模。Transformer参数可通过$M$个上下文示例$\{\textsf{C}_m\}_{m=1,M}$进行训练，训练后的模型随后应用于新的上下文数据$\textsf{C}_{M+1}$以处理新的$f_{M+1}(x)\in\textsf{F}$。目标是使Transformer能够为新查询$x_{N_{M+1}+1}$构建每个类别$c\in\{0,\dots,C-1\}$的概率。我们假设$f_m(x)$的每个分量存在于再生核希尔伯特空间（RKHS）中，从而定义了$\textsf{F}$。理论分析与大量实验表明，在前向传播过程中，Transformer（其注意力机制由RKHS核定义）实现了基础函数的某种梯度下降形式，该形式与softmax相关的隐向量函数相关联。我们首次在真实场景中展示了这种小样本学习方法的应用，该演示基于ImageNet数据集完成。