Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the [CLS] symbol and the tokens. We propose in this paper a simple modification that employs separate normalization layers for the tokens and the [CLS] symbol to better capture their distinct characteristics and enhance downstream task performance. Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles. We empirically show that by utilizing a separate normalization layer, the [CLS] embeddings can better encode the global contextual information and are distributed more uniformly in its anisotropic space. When replacing the conventional normalization layer with the two separate layers, we observe an average 2.7% performance improvement over the image, natural language, and graph domains.

翻译：自监督训练方法在Transformer模型上已在多个领域展现出卓越性能。以往基于Transformer的模型（如掩码自编码器MAE）通常对[CLS]符号和令牌使用单一归一化层。本文提出一种简单改进，即对令牌和[CLS]符号分别采用独立的归一化层，以更精准地捕捉两者差异特性并提升下游任务性能。本方法旨在缓解对两种令牌类型使用相同归一化统计量可能带来的负面效应——该方式可能无法与其各自角色最优匹配。实验表明，通过使用分离归一化层，[CLS]嵌入能更有效地编码全局上下文信息，并在其各向异性空间中分布更均匀。当将传统归一化层替换为两个独立层后，我们在图像、自然语言和图形领域平均观察到2.7%的性能提升。