Learning with large-scale unlabeled data has become a powerful tool for pre-training Visual Transformers (VTs). However, prior works tend to overlook that, in real-world scenarios, the input data may be corrupted and unreliable. Pre-training VTs on such corrupted data can be challenging, especially when we pre-train via the masked autoencoding approach, where both the inputs and masked ``ground truth" targets can potentially be unreliable in this case. To address this limitation, we introduce the Token Boosting Module (TBM) as a plug-and-play component for VTs that effectively allows the VT to learn to extract clean and robust features during masked autoencoding pre-training. We provide theoretical analysis to show how TBM improves model pre-training with more robust and generalizable representations, thus benefiting downstream tasks. We conduct extensive experiments to analyze TBM's effectiveness, and results on four corrupted datasets demonstrate that TBM consistently improves performance on downstream tasks.

翻译：利用大规模无标签数据进行学习已成为预训练视觉Transformer（VT）的强大工具。然而，现有工作往往忽略了一个事实：在现实场景中，输入数据可能受到污染且不可靠。在这种受损数据上预训练VT具有挑战性，尤其当我们通过掩码自编码方法进行预训练时，输入数据和掩码后的“真实标注”目标都可能不可靠。为解决这一局限，我们引入Token Boosting模块（TBM）作为VT的即插即用组件，有效使VT在掩码自编码预训练过程中学习提取干净且鲁棒的特征。我们提供理论分析，证明TBM如何通过更鲁棒且可泛化的表示改进模型预训练，从而惠及下游任务。我们进行了大量实验分析TBM的有效性，在四个受损数据集上的结果表明，TBM能持续提升下游任务的性能。