We propose InstructDET, a data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage are greatly diversified to encompass common user intentions related to object detection. For one image, we produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encompass common detection expressions, we involve emerging vision-language model (VLM) and large language model (LLM) to generate instructions guided by text prompts and object bbxs, as the generalizations of foundation models are effective to produce human-like expressions (e.g., describing object property, category, and relationship). We name our constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through using our InstructDET method. By using our InDET dataset, we show that a conventional ROD model surpasses existing methods on standard REC datasets and our InDET test set. Our data-centric method InstructDET, with automatic data expansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions.

翻译：我们提出InstructDET，一种以数据为中心的指代目标检测（ROD）方法，该方法根据用户指令定位目标物体。尽管源自指代表达（REC），但我们利用的指令被大幅多样化，以涵盖与目标检测相关的常见用户意图。对于单张图像，我们生成大量指令，这些指令可指代每个单独物体以及多个物体的不同组合。每条指令及其对应的目标边界框（bbxs）构成一个训练数据对。为了涵盖常见的检测表达，我们引入新兴的视觉-语言模型（VLM）和大语言模型（LLM），在文本提示和目标边界框引导下生成指令，因为基础模型的泛化能力能有效产生类人表达（如描述物体属性、类别和关系）。我们将构建的数据集命名为InDET。该数据集包含图像、边界框及来自基础模型的通用指令。InDET基于现有REC数据集和目标检测数据集开发，具有扩展潜力：任何带有物体边界框的图像均可通过使用我们的InstructDET方法纳入其中。通过使用InDET数据集，我们证明传统ROD模型在标准REC数据集和InDET测试集上均超越现有方法。这种以数据为中心的方法InstructDET，通过利用基础模型自动扩展数据，为ROD大幅多样化以执行常见目标检测指令开辟了有前景的研究方向。