We theoretically investigate whether the Query, Key, Value weight triplet can be reduced in encoder-only and decoder-only transformers. Under mild assumptions, we prove that Query weights are redundant and can be replaced with the identity matrix, reducing attention parameters by $25\%$. This also simplifies optimization: attention logits become linear rather than quadratic in learned weights. Validating on decoder-only GPT-style small models trained from scratch, we find that with adjusted attention scaling and weight decay, reduced models match baseline performance despite fewer parameters. Training remains stable at over $3\times$ lower weight decay, suggesting Query weight elimination provides implicit regularization. Our analysis has also led us to a structural expressivity boundary: in the mathematically tractable ReLU setting, skip connections push MLPs into a generically disjoint function class at fixed width. These findings motivate investigation across modalities and at scale, where the observed stability and efficiency gains may prove most consequential.

翻译：本研究从理论层面探讨了编码器专用与解码器专用Transformer中查询、键、值权重三元组的可简化性。在温和假设条件下，我们证明查询权重具有冗余性，可被单位矩阵替代，从而使注意力参数减少$25\%$。该简化同时优化了计算过程：注意力对数在可学习权重上呈现线性而非二次型特征。通过在解码器专用GPT架构的小型模型上进行从头训练的验证实验发现，经调整注意力缩放与权重衰减后，简化模型在参数更少的情况下仍能达到基线性能水平。训练过程在权重衰减降低超过$3$倍的条件下保持稳定，表明查询权重的消除提供了隐式正则化效应。本分析还揭示了结构表达能力边界：在数学可处理的ReLU设定中，跳跃连接将固定宽度的多层感知器推入泛型分离函数类。这些发现为跨模态与大规模场景的研究提供了新动力，其中观测到的稳定性与效率提升可能产生重要影响。