Recent self-supervised methods are mainly designed for representation learning with the base model, e.g., ResNets or ViTs. They cannot be easily transferred to DETR, with task-specific Transformer modules. In this work, we present Siamese DETR, a Siamese self-supervised pretraining approach for the Transformer architecture in DETR. We consider learning view-invariant and detection-oriented representations simultaneously through two complementary tasks, i.e., localization and discrimination, in a novel multi-view learning framework. Two self-supervised pretext tasks are designed: (i) Multi-View Region Detection aims at learning to localize regions-of-interest between augmented views of the input, and (ii) Multi-View Semantic Discrimination attempts to improve object-level discrimination for each region. The proposed Siamese DETR achieves state-of-the-art transfer performance on COCO and PASCAL VOC detection using different DETR variants in all setups. Code is available at https://github.com/Zx55/SiameseDETR.

翻译：近期设计的自监督方法主要面向基于基础模型（如ResNet或ViT）的表征学习，难以直接迁移至包含任务特定Transformer模块的DETR框架中。本文提出Siamese DETR，一种面向DETR中Transformer架构的孪生自监督预训练方法。我们通过新型多视角学习框架中的两项互补任务（定位与判别），同步学习视角不变性与检测导向的表征。具体设计了两项自监督预训练任务：(i) 多视角区域检测，旨在学习输入增强视图间感兴趣区域的定位；(ii) 多视角语义判别，致力于提升各区域的对象级判别能力。实验表明，所提出的Siamese DETR方法在COCO与PASCAL VOC检测数据集上，基于不同DETR变体的全配置下均达到最先进的迁移性能。代码开源于https://github.com/Zx55/SiameseDETR。