Few-shot object detection, which focuses on detecting novel objects with few labels, is an emerging challenge in the community. Recent studies show that adapting a pre-trained model or modified loss function can improve performance. In this paper, we explore leveraging the power of Contrastive Language-Image Pre-training (CLIP) and hard negative classification loss in low data setting. Specifically, we propose Re-scoring using Image-language Similarity for Few-shot object detection (RISF) which extends Faster R-CNN by introducing Calibration Module using CLIP (CM-CLIP) and Background Negative Re-scale Loss (BNRL). The former adapts CLIP, which performs zero-shot classification, to re-score the classification scores of a detector using image-class similarities, the latter is modified classification loss considering the punishment for fake backgrounds as well as confusing categories on a generalized few-shot object detection dataset. Extensive experiments on MS-COCO and PASCAL VOC show that the proposed RISF substantially outperforms the state-of-the-art approaches. The code will be available.

翻译：少样本目标检测旨在通过少量标注样本检测新类别目标，是该领域新兴的研究挑战。近期研究表明，采用预训练模型或改进损失函数可有效提升性能。本文探索了在低数据场景下利用对比语言-图像预训练（CLIP）与难负例分类损失的方法。具体而言，我们提出基于图像-语言相似度的少样本目标检测重评分方法（RISF），该方法通过引入CLIP校准模块（CM-CLIP）和背景负例重缩放损失（BNRL）扩展了Faster R-CNN框架。CM-CLIP模块利用CLIP的零样本分类能力，基于图像-类别相似度对检测器的分类分数进行重评分；BNRL损失函数针对广义少样本目标检测数据集，改进了分类损失中虚假背景与混淆类别的惩罚机制。在MS-COCO和PASCAL VOC数据集上的大量实验表明，所提出的RISF方法显著优于现有最优方法。相关代码将予以公开。