Referring Expression Segmentation (RES) and Comprehension (REC) respectively segment and detect the object described by an expression, while Referring Expression Generation (REG) generates an expression for the selected object. Existing datasets and methods commonly support single-target expressions only, i.e., one expression refers to one object, not considering multi-target and no-target expressions. This greatly limits the real applications of REx (RES/REC/REG). This paper introduces three new benchmarks called Generalized Referring Expression Segmentation (GRES), Comprehension (GREC), and Generation (GREG), collectively denoted as GREx, which extend the classic REx to allow expressions to identify an arbitrary number of objects. We construct the first large-scale GREx dataset gRefCOCO that contains multi-target, no-target, and single-target expressions and their corresponding images with labeled targets. GREx and gRefCOCO are designed to be backward-compatible with REx, facilitating extensive experiments to study the performance gap of the existing REx methods on GREx tasks. One of the challenges of GRES/GREC is complex relationship modeling, for which we propose a baseline ReLA that adaptively divides the image into regions with sub-instance clues and explicitly models the region-region and region-language dependencies. The proposed ReLA achieves the state-of-the-art results on the both GRES and GREC tasks. The proposed gRefCOCO dataset and method are available at https://henghuiding.github.io/GREx.

翻译：指代表达式分割（RES）与指代表达式理解（REC）分别对表达式所描述的目标进行分割与检测，而指代表达式生成（REG）则为选定目标生成描述表达式。现有数据集与方法通常仅支持单目标表达式，即一个表达式指代单个目标，未考虑多目标及无目标表达式。这极大限制了REx（RES/REC/REG）的实际应用。本文提出三个新基准任务：广义指代表达式分割（GRES）、广义指代表达式理解（GREC）与广义指代表达式生成（GREG），统称为GREx，将经典REx扩展至允许表达式指代任意数量目标。我们构建了首个大规模GREx数据集gRefCOCO，其中包含多目标、无目标及单目标表达式，以及带有标注目标的对应图像。GREx与gRefCOCO设计为向后兼容REx，便于通过大量实验研究现有REx方法在GREx任务上的性能差距。GRES/GREC的挑战之一在于复杂关系建模，为此我们提出基线方法ReLA，其自适应地将图像划分为包含子实例线索的区域，并显式建模区域-区域及区域-语言的依赖关系。所提出的ReLA在GRES与GREC任务上均取得了最先进的性能。gRefCOCO数据集与方法已在https://henghuiding.github.io/GREx发布。