Large language models (LLMs) have numerous real-life applications across various domains, such as natural language translation, sentiment analysis, language modeling, chatbots and conversational agents, creative writing, text classification, summarization, and generation. LLMs have shown great promise in improving the accuracy and efficiency of these tasks, and have the potential to revolutionize the field of natural language processing (NLP) in the years to come. Exponential function based attention unit is a fundamental element in LLMs. Several previous works have studied the convergence of exponential regression and softmax regression. The exponential regression [Li, Song, Zhou 2023] and softmax regression [Deng, Li, Song 2023] can be formulated as follows. Given matrix $A \in \mathbb{R}^{n \times d}$ and vector $b \in \mathbb{R}^n$, the goal of exponential regression is to solve \begin{align*} \min_{x} \| \exp(Ax) - b \|_2 \end{align*} and the goal of softmax regression is to solve \begin{align*} \min_{x} \| \langle \exp(Ax) , {\bf 1}_n \rangle^{-1} \exp(Ax) - b \|_2 . \end{align*} In this work, we define a slightly different formulation than softmax regression. \begin{align*} \min_{x \in \mathbb{R}^d } \| u(x) - \langle u(x) , {\bf 1}_n \rangle \cdot b \|_2 \end{align*} where $u(x) \in \{ \exp(Ax), \cosh(Ax) , \sinh(Ax) \}$. We provide an input sparsity time algorithm for this problem. Our algorithm framework is very general and can be applied to functions like $\cosh()$ and $\sinh()$ as well. Our technique is also general enough to be applied to in-context learning for rescaled softmax regression.

翻译：大型语言模型在自然语言翻译、情感分析、语言建模、聊天机器人与对话系统、创意写作、文本分类、摘要生成等众多领域具有广泛的实际应用。这些模型在提升上述任务精度与效率方面展现出巨大潜力，有望在未来几年内彻底改变自然语言处理领域。基于指数函数的注意力单元是大型语言模型的核心要素。已有若干工作研究了指数回归与Softmax回归的收敛性。指数回归[Li, Song, Zhou 2023]与Softmax回归[Deng, Li, Song 2023]可形式化描述如下：给定矩阵$A \in \mathbb{R}^{n \times d}$与向量$b \in \mathbb{R}^n$，指数回归的目标是求解\begin{align*} \min_{x} \| \exp(Ax) - b \|_2 \end{align*}，而Softmax回归的目标是求解\begin{align*} \min_{x} \| \langle \exp(Ax) , {\bf 1}_n \rangle^{-1} \exp(Ax) - b \|_2 \end{align*}。在本工作中，我们定义了一种与Softmax回归略有差异的形式：\begin{align*} \min_{x \in \mathbb{R}^d } \| u(x) - \langle u(x) , {\bf 1}_n \rangle \cdot b \|_2 \end{align*}，其中$u(x) \in \{ \exp(Ax), \cosh(Ax) , \sinh(Ax) \}$。针对该问题，我们提出了一种输入稀疏时间算法。该算法框架具有高度通用性，可适用于$\cosh()$与$\sinh()$等函数。此外，该技术还可推广至缩放Softmax回归的上下文学习场景。