Current remote-sensing interpretation models often focus on a single task such as detection, segmentation, or caption. However, the task-specific designed models are unattainable to achieve the comprehensive multi-level interpretation of images. The field also lacks support for multi-task joint interpretation datasets. In this paper, we propose Panoptic Perception, a novel task and a new fine-grained dataset (FineGrip) to achieve a more thorough and universal interpretation for RSIs. The new task, 1) integrates pixel-level, instance-level, and image-level information for universal image perception, 2) captures image information from coarse to fine granularity, achieving deeper scene understanding and description, and 3) enables various independent tasks to complement and enhance each other through multi-task learning. By emphasizing multi-task interactions and the consistency of perception results, this task enables the simultaneous processing of fine-grained foreground instance segmentation, background semantic segmentation, and global fine-grained image captioning. Concretely, the FineGrip dataset includes 2,649 remote sensing images, 12,054 fine-grained instance segmentation masks belonging to 20 foreground things categories, 7,599 background semantic masks for 5 stuff classes and 13,245 captioning sentences. Furthermore, we propose a joint optimization-based panoptic perception model. Experimental results on FineGrip demonstrate the feasibility of the panoptic perception task and the beneficial effect of multi-task joint optimization on individual tasks. The dataset will be publicly available.

翻译：当前遥感解译模型通常聚焦于检测、分割或描述等单一任务。然而，这类面向特定任务设计的模型无法实现对图像的多层次综合解译，且该领域缺乏支持多任务联合解译的数据集。本文提出全景感知这一新任务及其对应细粒度数据集FineGrip，旨在实现更全面、通用的遥感图像解译。该新任务：1）融合像素级、实例级和图像级信息实现通用图像感知；2）从粗到细的粒度捕捉图像信息，实现更深入的场景理解与描述；3）通过多任务学习使各独立任务相互补充增强。通过强调多任务交互与感知结果的一致性，该任务可同步处理细粒度前景实例分割、背景语义分割及全局细粒度图像描述。具体而言，FineGrip数据集包含2,649张遥感图像、12,054个细粒度实例分割掩膜（覆盖20个前景物体类别）、7,599个背景语义分割掩膜（涵盖5个背景类别）及13,245条描述语句。此外，本文提出基于联合优化的全景感知模型。在FineGrip上的实验结果表明全景感知任务的可行性以及多任务联合优化对单个任务的促进作用。该数据集将公开提供。