Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers, where performance worsens as the number of layers increases. Instead of characterizing oversmoothing from the view of complete collapse in which representations converge to a single point, we dive into a more general perspective of dimensional collapse in which representations lie in a narrow cone. Accordingly, inspired by the effectiveness of contrastive learning in preventing dimensional collapse, we propose a novel normalization layer called ContraNorm. Intuitively, ContraNorm implicitly shatters representations in the embedding space, leading to a more uniform distribution and a slighter dimensional collapse. On the theoretical analysis, we prove that ContraNorm can alleviate both complete collapse and dimensional collapse under certain conditions. Our proposed normalization layer can be easily integrated into GNNs and Transformers with negligible parameter overhead. Experiments on various real-world datasets demonstrate the effectiveness of our proposed ContraNorm. Our implementation is available at https://github.com/PKU-ML/ContraNorm.

翻译：过平滑是图神经网络（GNNs）与Transformer中普遍存在的现象，表现为随着网络层数增加模型性能下降。不同于将过平滑视为表征收敛至单一点的完全坍缩，我们从更广义的维度坍缩视角切入——即表征局限于狭窄锥形空间。受对比学习在抑制维度坍缩方面成效的启发，我们提出新型归一化层ContraNorm。直观而言，ContraNorm能隐式打散嵌入空间中的表征，促使其分布更均匀并减轻维度坍缩。理论分析证明，ContraNorm在特定条件下可同时缓解完全坍缩与维度坍缩。该归一化层可轻松集成至GNNs与Transformer中，且参数量可忽略不计。多组真实数据集实验验证了ContraNorm的有效性。相关实现代码已开源至https://github.com/PKU-ML/ContraNorm。