Large language models (LLMs) have brought significant and transformative changes in human society. These models have demonstrated remarkable capabilities in natural language understanding and generation, leading to various advancements and impacts across several domains. We consider the in-context learning under two formulation for attention related regression in this work. Given matrices $A_1 \in \mathbb{R}^{n \times d}$, and $A_2 \in \mathbb{R}^{n \times d}$ and $B \in \mathbb{R}^{n \times n}$, the purpose is to solve some certain optimization problems: Normalized version $\min_{X} \| D(X)^{-1} \exp(A_1 X A_2^\top) - B \|_F^2$ and Rescaled version $\| \exp(A_1 X A_2^\top) - D(X) \cdot B \|_F^2$. Here $D(X) := \mathrm{diag}( \exp(A_1 X A_2^\top) {\bf 1}_n )$. Our regression problem shares similarities with previous studies on softmax-related regression. Prior research has extensively investigated regression techniques related to softmax regression: Normalized version $\| \langle \exp(Ax) , {\bf 1}_n \rangle^{-1} \exp(Ax) - b \|_2^2$ and Resscaled version $\| \exp(Ax) - \langle \exp(Ax), {\bf 1}_n \rangle b \|_2^2 $ In contrast to previous approaches, we adopt a vectorization technique to address the regression problem in matrix formulation. This approach expands the dimension from $d$ to $d^2$, resembling the formulation of the regression problem mentioned earlier. Upon completing the lipschitz analysis of our regression function, we have derived our main result concerning in-context learning.

翻译：大语言模型已在人类社会带来重大变革性影响。这类模型在自然语言理解与生成方面展现出卓越能力，进而推动多个领域的进步与变革。本研究探讨两种注意力相关回归框架下的上下文学习问题。给定矩阵$A_1 \in \mathbb{R}^{n \times d}$、$A_2 \in \mathbb{R}^{n \times d}$与$B \in \mathbb{R}^{n \times n}$，目标在于求解特定优化问题：归一化版本$\min_{X} \| D(X)^{-1} \exp(A_1 X A_2^\top) - B \|_F^2$ 与重缩放版本$\| \exp(A_1 X A_2^\top) - D(X) \cdot B \|_F^2$，其中$D(X) := \mathrm{diag}( \exp(A_1 X A_2^\top) {\bf 1}_n )$。我们的回归问题与先前关于softmax回归的研究存在相似性。已有研究深入探讨了softmax回归相关技术：归一化版本$\| \langle \exp(Ax) , {\bf 1}_n \rangle^{-1} \exp(Ax) - b \|_2^2$与重缩放版本$\| \exp(Ax) - \langle \exp(Ax), {\bf 1}_n \rangle b \|_2^2 $。与以往方法不同，本研究采用向量化技术处理矩阵形式的回归问题。该方法将维度从$d$扩展至$d^2$，形式上与前述回归问题表述相似。完成回归函数的利普希茨分析后，我们推导出关于上下文学习的主要结论。