We propose InstructDET, a data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage are greatly diversified to encompass common user intentions related to object detection. For one image, we produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encompass common detection expressions, we involve emerging vision-language model (VLM) and large language model (LLM) to generate instructions guided by text prompts and object bbxs, as the generalizations of foundation models are effective to produce human-like expressions (e.g., describing object property, category, and relationship). We name our constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through using our InstructDET method. By using our InDET dataset, we show that a conventional ROD model surpasses existing methods on standard REC datasets and our InDET test set. Our data-centric method InstructDET, with automatic data expansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions.

翻译：我们提出InstructDET，一种以数据为中心的方法用于指代目标检测（ROD），该方法根据用户指令定位目标对象。虽然源于指代表达（REC），但我们利用的指令经过大幅多样化，以涵盖与目标检测相关的常见用户意图。对于一张图像，我们生成海量指令，指向每个单独对象及不同组合的多对象。每条指令及其对应的目标边界框（bbxs）构成一个训练数据对。为涵盖常见检测表达，我们引入新兴视觉语言模型（VLM）和大语言模型（LLM），在文本提示和目标边界框引导下生成指令，利用基础模型的泛化能力生成类人表达（例如描述对象属性、类别及关系）。我们将构建的数据集命名为InDET。它包含来自基础模型的图像、边界框及广义指令。我们的InDET基于现有REC数据集与目标检测数据集开发，并具有扩展潜力：任何带有对象边界框的图像均可通过InstructDET方法纳入。通过使用InDET数据集，我们证明传统ROD模型在标准REC数据集及InDET测试集上均超越现有方法。我们以数据为中心的InstructDET方法，借助基础模型自动扩展数据，开辟了ROD可大幅多样化以执行常见目标检测指令的前景领域。