The use of self-supervised pre-training has emerged as a promising approach to enhance the performance of visual tasks such as image classification. In this context, recent approaches have employed the Masked Image Modeling paradigm, which pre-trains a backbone by reconstructing visual tokens associated with randomly masked image patches. This masking approach, however, introduces noise into the input data during pre-training, leading to discrepancies that can impair performance during the fine-tuning phase. Furthermore, input masking neglects the dependencies between corrupted patches, increasing the inconsistencies observed in downstream fine-tuning tasks. To overcome these issues, we propose a new self-supervised pre-training approach, named Masked and Permuted Vision Transformer (MaPeT), that employs autoregressive and permuted predictions to capture intra-patch dependencies. In addition, MaPeT employs auxiliary positional information to reduce the disparity between the pre-training and fine-tuning phases. In our experiments, we employ a fair setting to ensure reliable and meaningful comparisons and conduct investigations on multiple visual tokenizers, including our proposed $k$-CLIP which directly employs discretized CLIP features. Our results demonstrate that MaPeT achieves competitive performance on ImageNet, compared to baselines and competitors under the same model setting. Source code and trained models are publicly available at: https://github.com/aimagelab/MaPeT.

翻译：自监督预训练作为一种增强图像分类等视觉任务性能的方法已崭露头角。在此背景下，近期方法采用了掩码图像建模范式，该范式通过重建与随机掩码图像块相关联的视觉标记来预训练主干网络。然而，这种掩码方法在预训练期间会向输入数据引入噪声，导致差异并损害微调阶段的性能。此外，输入掩码忽略了被破坏图像块之间的依赖关系，增加了下游微调任务中观察到的不一致性。为解决这些问题，我们提出了一种新的自监督预训练方法——掩码与排列视觉Transformer（MaPeT），该方法采用自回归和排列预测来捕获图像块内部依赖关系。此外，MaPeT利用辅助位置信息来减少预训练与微调阶段之间的差异。在实验中，我们采用公平设置确保可靠且有意义的比较，并对多种视觉标记器进行了研究，包括我们提出的$k$-CLIP（该标记器直接利用离散化的CLIP特征）。结果表明，在相同模型设置下，与基线及竞争方法相比，MaPeT在ImageNet上取得了具有竞争力的性能。源代码与预训练模型已公开于：https://github.com/aimagelab/MaPeT。