The self-attention mechanism has emerged as a critical component for improving the performance of various backbone neural networks. However, current mainstream approaches individually incorporate newly designed self-attention modules (SAMs) into each layer of the network for granted without fully exploiting their parameters' potential. This leads to suboptimal performance and increased parameter consumption as the network depth increases. To improve this paradigm, in this paper, we first present a counterintuitive but inherent phenomenon: SAMs tend to produce strongly correlated attention maps across different layers, with an average Pearson correlation coefficient of up to 0.85. Inspired by this inherent observation, we propose Dense-and-Implicit Attention (DIA), which directly shares SAMs across layers and employs a long short-term memory module to calibrate and bridge the highly correlated attention maps of different layers, thus improving the parameter utilization efficiency of SAMs. This design of DIA is also consistent with the neural network's dynamical system perspective. Through extensive experiments, we demonstrate that our simple yet effective DIA can consistently enhance various network backbones, including ResNet, Transformer, and UNet, across tasks such as image classification, object detection, and image generation using diffusion models.

翻译：自注意力机制已成为提升多种骨干神经网络性能的关键组件。然而，当前主流方法通常将新设计的自注意力模块（SAM）直接嵌入网络的每一层，并未充分利用其参数潜力。这导致随着网络深度的增加，性能欠优且参数消耗增大。为改进这一范式，本文首先揭示了一个反直觉但固有的现象：SAM在不同层间会产生高度相关的注意力图，平均皮尔逊相关系数高达0.85。受此固有观察启发，我们提出密集隐式注意力（DIA），该方法跨层直接共享SAM，并采用长短期记忆模块来校准和桥接不同层间高度相关的注意力图，从而提升SAM的参数利用效率。DIA的设计也与神经网络的动力系统视角相一致。通过大量实验，我们证明这种简单而有效的DIA能够持续增强多种网络骨干结构（包括ResNet、Transformer和UNet），并在图像分类、目标检测以及使用扩散模型的图像生成等任务上取得一致效果。