Despite considerable progress, the advancement of Panoptic Narrative Grounding (PNG) remains hindered by costly annotations. In this paper, we introduce a novel Semi-Supervised Panoptic Narrative Grounding (SS-PNG) learning scheme, capitalizing on a smaller set of labeled image-text pairs and a larger set of unlabeled pairs to achieve competitive performance. Unlike visual segmentation tasks, PNG involves one pixel belonging to multiple open-ended nouns. As a result, existing multi-class based semi-supervised segmentation frameworks cannot be directly applied to this task. To address this challenge, we first develop a novel SS-PNG Network (SS-PNG-NW) tailored to the SS-PNG setting. We thoroughly investigate strategies such as Burn-In and data augmentation to determine the optimal generic configuration for the SS-PNG-NW. Additionally, to tackle the issue of imbalanced pseudo-label quality, we propose a Quality-Based Loss Adjustment (QLA) approach to adjust the semi-supervised objective, resulting in an enhanced SS-PNG-NW+. Employing our proposed QLA, we improve BCE Loss and Dice loss at pixel and mask levels, respectively. We conduct extensive experiments on PNG datasets, with our SS-PNG-NW+ demonstrating promising results comparable to fully-supervised models across all data ratios. Remarkably, our SS-PNG-NW+ outperforms fully-supervised models with only 30% and 50% supervision data, exceeding their performance by 0.8% and 1.1% respectively. This highlights the effectiveness of our proposed SS-PNG-NW+ in overcoming the challenges posed by limited annotations and enhancing the applicability of PNG tasks. The source code is available at https://github.com/nini0919/SSPNG.

翻译：尽管取得了显著进展，但全景叙事定位（PNG）的进步仍受限于高昂的标注成本。本文提出了一种新型半监督全景叙事定位（SS-PNG）学习方案，利用少量标注的图像-文本对和大量未标注对，实现具有竞争力的性能。与视觉分割任务不同，PNG涉及一个像素可能属于多个开放式名词。因此，现有的基于多类别的半监督分割框架无法直接应用于此任务。为应对这一挑战，我们首先开发了一种专门针对SS-PNG设定而设计的新型SS-PNG网络（SS-PNG-NW）。我们深入研究了Burn-In和数据增强等策略，以确定SS-PNG-NW的最优通用配置。此外，为解决伪标签质量不平衡的问题，我们提出了一种基于质量的损失调整（QLA）方法，用于调整半监督目标函数，从而得到增强版SS-PNG-NW+。通过采用提出的QLA，我们分别在像素级和掩码级改进了BCE损失和Dice损失。我们在PNG数据集上进行了大量实验，结果表明，我们的SS-PNG-NW+在所有数据比例下均能取得与全监督模型相当且令人满意的结果。值得注意的是，仅使用30%和50%的监督数据时，我们的SS-PNG-NW+性能分别超过全监督模型0.8%和1.1%。这凸显了我们提出的SS-PNG-NW+在克服有限标注带来的挑战以及提升PNG任务适用性方面的有效性。源代码可在https://github.com/nini0919/SSPNG获取。