Standard softmax self-attention excels in vision tasks but incurs quadratic complexity O(N^2), limiting high-resolution deployment. Linear attention reduces the cost to O(N), yet its compressed state representations can impair modeling capacity and accuracy. We present an analytical study that contrasts linear and softmax attention for visual representation learning from a layer-stacking perspective. We further conduct systematic experiments on layer-wise hybridization patterns of linear and softmax attention. Our results show that, compared with rigid intra-block hybrid designs, fine-grained layer-wise hybridization can match or surpass performance while requiring fewer softmax layers. Building on these findings, we propose SoLA-Vision (Softmax-Linear Attention Vision), a flexible layer-wise hybrid attention backbone that enables fine-grained control over how linear and softmax attention are integrated. By strategically inserting a small number of global softmax layers, SoLA-Vision achieves a strong trade-off between accuracy and computational cost. On ImageNet-1K, SoLA-Vision outperforms purely linear and other hybrid attention models. On dense prediction tasks, it consistently surpasses strong baselines by a considerable margin. Code will be released.

翻译：标准softmax自注意力机制在视觉任务中表现出色，但其二次复杂度O(N^2)限制了在高分辨率场景下的部署。线性注意力将计算成本降至O(N)，但其压缩的状态表示可能损害建模能力和精度。我们从层堆叠的视角出发，对线性和softmax注意力在视觉表示学习中的作用进行了对比分析。我们进一步对线性和softmax注意力的逐层混合模式进行了系统性实验。结果表明，与僵化的块内混合设计相比，细粒度的逐层混合方案在减少softmax层使用的同时，能够达到甚至超越其性能。基于这些发现，我们提出了SoLA-Vision（Softmax-Linear Attention Vision），一种灵活的逐层混合注意力骨干网络，能够对线性和softmax注意力的融合方式进行细粒度控制。通过策略性地插入少量全局softmax层，SoLA-Vision在精度与计算成本之间实现了优异的权衡。在ImageNet-1K数据集上，SoLA-Vision的性能优于纯线性及其他混合注意力模型。在密集预测任务中，其表现持续显著超越强基线方法。代码将公开。