We propose InstructDET, a data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage are greatly diversified to encompass common user intentions related to object detection. For one image, we produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encompass common detection expressions, we involve emerging vision-language model (VLM) and large language model (LLM) to generate instructions guided by text prompts and object bbxs, as the generalizations of foundation models are effective to produce human-like expressions (e.g., describing object property, category, and relationship). We name our constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through using our InstructDET method. By using our InDET dataset, we show that a conventional ROD model surpasses existing methods on standard REC datasets and our InDET test set. Our data-centric method InstructDET, with automatic data expansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions.

翻译：我们提出InstructDET，一种以数据为中心的指代目标检测（ROD）方法，该方法根据用户指令定位目标对象。虽然源自指代表达（REC），但我们利用的指令经过大幅多样化，涵盖了与目标检测相关的常见用户意图。对于一张图像，我们生成海量指令，指向每一个单独对象以及多个对象的不同组合。每条指令及其对应的目标边界框（bbxs）构成一个训练数据对。为了涵盖常见的检测表达，我们引入新兴的视觉语言模型（VLM）和大语言模型（LLM），在文本提示和目标边界框的引导下生成指令，因为基础模型的泛化能力能有效产生类人表达（例如，描述对象属性、类别和关系）。我们将构建的数据集命名为InDET。它包含来自基础模型的图像、边界框和泛化指令。我们的InDET基于现有REC数据集和目标检测数据集开发，并具有扩展潜力：任何包含对象边界框的图像均可通过使用我们的InstructDET方法纳入。通过使用InDET数据集，我们证明传统的ROD模型在标准REC数据集和InDET测试集上均超越了现有方法。我们以数据为中心的InstructDET方法，通过利用基础模型实现自动数据扩展，开辟了一个有前景的研究方向：ROD可以大幅多样化，以执行常见的目标检测指令。