Large language models (LLMs) have made transformed changes for human society. One of the key computation in LLMs is the softmax unit. This operation is important in LLMs because it allows the model to generate a distribution over possible next words or phrases, given a sequence of input words. This distribution is then used to select the most likely next word or phrase, based on the probabilities assigned by the model. The softmax unit plays a crucial role in training LLMs, as it allows the model to learn from the data by adjusting the weights and biases of the neural network. In the area of convex optimization such as using central path method to solve linear programming. The softmax function has been used a crucial tool for controlling the progress and stability of potential function [Cohen, Lee and Song STOC 2019, Brand SODA 2020]. In this work, inspired the softmax unit, we define a softmax regression problem. Formally speaking, given a matrix $A \in \mathbb{R}^{n \times d}$ and a vector $b \in \mathbb{R}^n$, the goal is to use greedy type algorithm to solve \begin{align*} \min_{x} \| \langle \exp(Ax), {\bf 1}_n \rangle^{-1} \exp(Ax) - b \|_2^2. \end{align*} In certain sense, our provable convergence result provides theoretical support for why we can use greedy algorithm to train softmax function in practice.

翻译：大语言模型已经为人类社会带来了变革性变化。大语言模型中的关键计算之一便是softmax单元。该操作在大语言模型中具有重要意义，因为它能使模型在给定输入词序列的情况下，为可能的下一个词或短语生成概率分布。随后，该分布将根据模型赋予的概率，用于选择最可能的下一个词或短语。softmax单元在大语言模型训练中扮演着关键角色，它允许模型通过调整神经网络的权重和偏置来从数据中学习。在凸优化领域（例如使用中心路径法求解线性规划），softmax函数已被作为控制势函数进展与稳定性的重要工具 [Cohen, Lee and Song STOC 2019, Brand SODA 2020]。受softmax单元的启发，本文定义了一个softmax回归问题。形式化地，给定矩阵 $A \in \mathbb{R}^{n \times d}$ 和向量 $b \in \mathbb{R}^n$，目标是使用贪心类算法求解 \begin{align*} \min_{x} \| \langle \exp(Ax), {\bf 1}_n \rangle^{-1} \exp(Ax) - b \|_2^2。 \end{align*} 在某种意义上，我们的可证明收敛结果从理论上支持了为何在实践中可以使用贪心算法训练softmax函数。