Masked Image Modeling (MIM) is a new self-supervised vision pre-training paradigm using Vision Transformer (ViT). Previous works can be pixel-based or token-based, using original pixels or discrete visual tokens from parametric tokenizer models, respectively. Our proposed approach, \textbf{CCViT}, leverages k-means clustering to obtain centroids for image modeling without supervised training of tokenizer model. The centroids represent patch pixels and index tokens and have the property of local invariance. Non-parametric centroid tokenizer only takes seconds to create and is faster for token inference. Specifically, we adopt patch masking and centroid replacement strategies to construct corrupted inputs, and two stacked encoder blocks to predict corrupted patch tokens and reconstruct original patch pixels. Experiments show that the ViT-B model with only 300 epochs achieves 84.3\% top-1 accuracy on ImageNet-1K classification and 51.6\% on ADE20K semantic segmentation. Our approach achieves competitive results with BEiTv2 without distillation training from other models and outperforms other methods such as MAE.

翻译：掩码图像建模（Masked Image Modeling, MIM）是一种利用视觉Transformer（Vision Transformer, ViT）的新型自监督视觉预训练范式。先前的工作可分为像素级或标记级方法，分别使用原始像素或来自参数化分词器模型的离散视觉标记。我们提出的方法，即\textbf{CCViT}，利用k-means聚类获取质心，无需对分词器模型进行监督训练即可进行图像建模。这些质心代表图像块的像素和索引标记，并具有局部不变性。非参数化质心分词器只需数秒即可创建，且标记推理速度更快。具体而言，我们采用图像块掩码和质心替换策略构建受损输入，并通过两个堆叠的编码器模块预测受损图像块标记并重建原始图像块像素。实验表明，仅需300个训练周期，ViT-B模型在ImageNet-1K分类任务上达到84.3%的top-1准确率，在ADE20K语义分割任务上达到51.6%的mIoU。我们的方法在无需从其他模型进行蒸馏训练的情况下，达到了与BEiTv2相竞争的结果，并优于MAE等其他方法。