Self-supervised learning (SSL) trains on inputs alone, allowing us to make models that are more useful in more settings. The autoencoder SSL framework learns by reconstructing its own input after information loss through a bottleneck or noise injection. Masked autoencoders (MAE) are the most successful instantiation of this framework: they encode a random subset of patches, then decode the masked-out patches. In this work, we introduce key modifications to improve MAEs. Our method augments an image in two different ways, then masks and encodes each view separately. It then exchanges the global representations (CLS tokens) between views before decoding the masked patches. By design, our Masked Swingers encourages learning a view-agnostic summary of the image for efficient transfer. We perform extensive experiments, and find Masked Swingers outperforms the official MAE checkpoint by +10.7% averaged over 10 frozen-feature tasks at equal pre-training compute. Gains are highest on fine-grained tasks, e.g., instance retrieval. Swingers also reduces error by 64% relative to MAE on three new state-probing datasets for world modeling, while keeping the fine-tunability of MAE. Post-training by swinging nearest neighbors gains again, e.g., by +5% on IN-1K kNN, opening the door to learning invariances via autoencoding.
翻译:暂无翻译