In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean $26.1$ AP. Code will be available at \url{https://github.com/IDEA-Research/GroundingDINO}.

翻译：本文提出了一种名为Grounding DINO的开放集目标检测器，通过将基于Transformer的检测器DINO与接地预训练相结合，实现对任意目标的检测，支持类别名称或指代表达等人类输入。开放集目标检测的关键在于将语言引入闭集检测器以实现开放集概念泛化。为有效融合语言与视觉模态，我们从概念上将闭集检测器划分为三个阶段，并提出一种紧密融合方案，包含特征增强器、语言引导的查询选择以及用于跨模态融合的跨模态解码器。以往研究主要针对新类别评估开放集目标检测性能，而本文进一步提出在指定属性目标的指代表达理解上开展评估。Grounding DINO在三种设置下均表现优异，涵盖COCO、LVIS、ODinW及RefCOCO/+/g基准测试。在无需任何COCO训练数据的情况下，Grounding DINO在COCO检测零样本迁移基准上达到52.5 AP，同时在ODinW零样本基准上以平均26.1 AP创下新纪录。代码将发布于\url{https://github.com/IDEA-Research/GroundingDINO}。