In recent research, significant attention has been devoted to the open-vocabulary object detection task, aiming to generalize beyond the limited number of classes labeled during training and detect objects described by arbitrary category names at inference. Compared with conventional object detection, open vocabulary object detection largely extends the object detection categories. However, it relies on calculating the similarity between image regions and a set of arbitrary category names with a pretrained vision-and-language model. This implies that, despite its open-set nature, the task still needs the predefined object categories during the inference stage. This raises the question: What if we do not have exact knowledge of object categories during inference? In this paper, we call such a new setting as generative open-ended object detection, which is a more general and practical problem. To address it, we formulate object detection as a generative problem and propose a simple framework named GenerateU, which can detect dense objects and generate their names in a free-form way. Particularly, we employ Deformable DETR as a region proposal generator with a language model translating visual regions to object names. To assess the free-form object detection task, we introduce an evaluation method designed to quantitatively measure the performance of generative outcomes. Extensive experiments demonstrate strong zero-shot detection performance of our GenerateU. For example, on the LVIS dataset, our GenerateU achieves comparable results to the open-vocabulary object detection method GLIP, even though the category names are not seen by GenerateU during inference. Code is available at: https:// github.com/FoundationVision/GenerateU .

翻译：近年来的研究聚焦于开放词汇目标检测任务，旨在突破训练阶段有限类别标注的限制，在推理时检测任意类别名称描述的物体。与常规目标检测相比，开放词汇目标检测大幅扩展了检测类别范围，但其依赖预训练的视觉-语言模型计算图像区域与任意类别名称集合的相似度。这意味着，尽管具有开放集特性，该任务在推理阶段仍需预定义目标类别。这引出一个问题：若在推理时无法确切获知目标类别，该如何应对？本文将此新设定称为生成式开放式目标检测，它更具一般性和实用性。为解决该问题，我们将目标检测建模为生成式任务，提出名为GenerateU的简洁框架，可检测密集目标并自由形式生成其名称。具体而言，我们采用Deformable DETR作为区域提议生成器，配合语言模型将视觉区域转化为目标名称。为评估自由形式目标检测任务，我们引入一种专门量化生成结果性能的评估方法。大量实验表明，GenerateU展现出强大的零样本检测性能。例如，在LVIS数据集上，即使推理时未见类别名称，GenerateU仍取得了与开放词汇目标检测方法GLIP相当的结果。代码已开源至：https://github.com/FoundationVision/GenerateU