We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a contrastive image-text pretraining recipe to bridge the gap between image-level pretraining and open-vocabulary object detection. At the pretraining phase, we propose to randomly crop and resize regions of positional embeddings instead of using the whole image positional embeddings. This better matches the use of positional embeddings at region-level in the detection finetuning phase. In addition, we replace the common softmax cross entropy loss in contrastive learning with focal loss to better learn the informative yet difficult examples. Finally, we leverage recent advances in novel object proposals to improve open-vocabulary detection finetuning. We evaluate our full model on the LVIS and COCO open-vocabulary detection benchmarks and zero-shot transfer. RO-ViT achieves a state-of-the-art 32.1 $AP_r$ on LVIS, surpassing the best existing approach by +5.8 points in addition to competitive zero-shot transfer detection. Surprisingly, RO-ViT improves the image-level representation as well and achieves the state of the art on 9 out of 12 metrics on COCO and Flickr image-text retrieval benchmarks, outperforming competitive approaches with larger models.

翻译：我们提出区域感知开放词汇视觉Transformer（RO-ViT）——一种对比图像-文本预训练方案，旨在弥合图像级预训练与开放词汇目标检测之间的差距。在预训练阶段，我们提出随机裁剪并调整位置编码区域，而非使用整张图像的位置编码。这能更好地匹配检测微调阶段在区域级别使用位置编码的方式。此外，我们在对比学习中用焦点损失替代常用的softmax交叉熵损失，以更有效地学习信息丰富但困难的样本。最后，我们利用新型目标提议的最新进展改进开放词汇检测微调。我们在LVIS和COCO开放词汇检测基准及零样本迁移任务上评估完整模型。RO-ViT在LVIS上实现了32.1% $AP_r$的最优性能，超越现有最佳方法5.8个百分点，同时具备竞争力的零样本迁移检测性能。令人惊讶的是，RO-ViT还提升了图像级表示能力，在COCO和Flickr图像-文本检索基准的12项指标中取得9项最优，性能优于使用更大模型的竞争方法。