We present a novel masked image modeling (MIM) approach, context autoencoder (CAE), for self-supervised representation pretraining. We pretrain an encoder by making predictions in the encoded representation space. The pretraining tasks include two tasks: masked representation prediction - predict the representations for the masked patches, and masked patch reconstruction - reconstruct the masked patches. The network is an encoder-regressor-decoder architecture: the encoder takes the visible patches as input; the regressor predicts the representations of the masked patches, which are expected to be aligned with the representations computed from the encoder, using the representations of visible patches and the positions of visible and masked patches; the decoder reconstructs the masked patches from the predicted encoded representations. The CAE design encourages the separation of learning the encoder (representation) from completing the pertaining tasks: masked representation prediction and masked patch reconstruction tasks, and making predictions in the encoded representation space empirically shows the benefit to representation learning. We demonstrate the effectiveness of our CAE through superior transfer performance in downstream tasks: semantic segmentation, object detection and instance segmentation, and classification. The code will be available at https://github.com/Atten4Vis/CAE.

翻译：我们提出了一种新颖的掩码图像建模（MIM）方法——上下文自编码器（CAE），用于自监督表示预训练。通过在编码表示空间中进行预测，我们对编码器进行预训练。预训练任务包括两个：掩码表示预测——预测掩码补丁的表示，以及掩码补丁重建——重建掩码补丁。网络采用编码器-回归器-解码器架构：编码器以可见补丁作为输入；回归器利用可见补丁的表示以及可见补丁和掩码补丁的位置，预测掩码补丁的表示，这些表示需与编码器计算得到的表示对齐；解码器从预测的编码表示重建掩码补丁。CAE的设计鼓励将编码器（表示）学习与完成预训练任务（掩码表示预测和掩码补丁重建）分离，且实证表明在编码表示空间中进行预测对表示学习有益。我们通过在下游任务（语义分割、目标检测、实例分割和分类）中优越的迁移性能证明了CAE的有效性。代码将发布于https://github.com/Atten4Vis/CAE。