We study gradient flow on the exponential loss for a classification problem with a one-layer softmax attention model, where the key and query weight matrices are trained separately. Under a separability assumption on the data, we show that when gradient flow achieves the minimal loss value, it further implicitly minimizes the nuclear norm of the product of the key and query weight matrices. Such implicit regularization can be described by a Support Vector Machine (SVM) problem with respect to the attention weights. This finding contrasts with prior results showing that the gradient descent induces an implicit regularization on the Frobenius norm on the product weight matrix when the key and query matrices are combined into a single weight matrix for training. For diagonal key and query matrices, our analysis builds upon the reparameterization technique and exploits approximate KKT conditions of the SVM associated with the classification data. Moreover, the results are extended to general weights configurations given proper alignment of the weight matrices' singular spaces with the data features at initialization.

翻译：我们研究了分类问题中采用单层softmax注意力模型时的指数损失梯度流，其中键矩阵和查询矩阵分别独立训练。在数据满足可分性假设的条件下，我们证明当梯度流达到最小损失值时，会进一步隐式地最小化键矩阵与查询矩阵乘积的核范数。这种隐式正则化可通过关于注意力权重的支持向量机（SVM）问题进行描述。这一发现与先前的研究结果形成对比：当键矩阵与查询矩阵合并为单个权重矩阵进行训练时，梯度下降会诱导乘积权重矩阵的弗罗贝尼乌斯范数的隐式正则化。针对对角键矩阵与查询矩阵情形，我们的分析基于重参数化技术，并利用分类数据相关支持向量机（SVM）的近似KKT条件。此外，在初始化时权重矩阵奇异空间与数据特征呈现适当对齐的情况下，相关结论可推广至一般权重配置。