We propose InstructDET, a data-centric method for referring object detection (ROD) that localizes target objects based on user instructions. While deriving from referring expressions (REC), the instructions we leverage are greatly diversified to encompass common user intentions related to object detection. For one image, we produce tremendous instructions that refer to every single object and different combinations of multiple objects. Each instruction and its corresponding object bounding boxes (bbxs) constitute one training data pair. In order to encompass common detection expressions, we involve emerging vision-language model (VLM) and large language model (LLM) to generate instructions guided by text prompts and object bbxs, as the generalizations of foundation models are effective to produce human-like expressions (e.g., describing object property, category, and relationship). We name our constructed dataset as InDET. It contains images, bbxs and generalized instructions that are from foundation models. Our InDET is developed from existing REC datasets and object detection datasets, with the expanding potential that any image with object bbxs can be incorporated through using our InstructDET method. By using our InDET dataset, we show that a conventional ROD model surpasses existing methods on standard REC datasets and our InDET test set. Our data-centric method InstructDET, with automatic data expansion by leveraging foundation models, directs a promising field that ROD can be greatly diversified to execute common object detection instructions.

翻译：我们提出InstructDET，一种以数据为中心的指代目标检测（ROD）方法，该方法可根据用户指令定位目标物体。尽管源自指代表达（REC），我们所利用的指令高度多样化，涵盖与目标检测相关的常见用户意图。对于一张图像，我们生成海量指令，这些指令指代单个物体以及多个物体的不同组合。每条指令及其对应的目标边界框（bbxs）构成一个训练数据对。为了涵盖常见的检测表达，我们引入新兴的视觉语言模型（VLM）和大语言模型（LLM），在文本提示和目标边界框引导下生成指令——因为基础模型的泛化能力能有效产生类人表达（例如描述物体属性、类别和关系）。我们将构建的数据集命名为InDET。它包含图像、边界框以及来自基础模型的广义指令。我们的InDET基于已有REC数据集和目标检测数据集开发，并具备扩展潜力：任何包含物体边界框的图像均可通过使用我们的InDET方法融入其中。通过使用我们的InDET数据集，我们证明传统ROD模型在标准REC数据集和我们的InDET测试集上均超越现有方法。以数据为中心的InDET方法，通过利用基础模型实现自动数据扩展，开辟了一个有前景的研究方向：即ROD可被极大多样化以执行常见目标检测指令。