The self-attention mechanism in transformers and the message-passing mechanism in graph neural networks are repeatedly applied within deep learning architectures. We show that this application inevitably leads to oversmoothing, i.e., to similar representations at the deeper layers for different tokens in transformers and different nodes in graph neural networks. Based on our analysis, we present a correction term to the aggregating operator of these mechanisms. Empirically, this simple term eliminates much of the oversmoothing problem in visual transformers, obtaining performance in weakly supervised segmentation that surpasses elaborate baseline methods that introduce multiple auxiliary networks and training phrases. In graph neural networks, the correction term enables the training of very deep architectures more effectively than many recent solutions to the same problem.

翻译：Transformer中的自注意力机制与图神经网络中的消息传递机制在深度学习架构中被反复应用。我们证明，这种反复应用必然会导致过平滑现象，即Transformer中不同词元以及图神经网络中不同节点在深层网络中产生相似的表征。基于这一分析，我们提出对这些机制的聚合算子进行修正。实验表明，这一简洁的修正项能大幅缓解视觉Transformer中的过平滑问题，在弱监督分割任务上的表现超越了引入多个辅助网络与训练阶段的复杂基线方法。在图神经网络中，该修正项相比现有同类问题的诸多解决方案，能够更高效地训练极深层的网络架构。