We present a novel data-efficient semi-supervised framework to improve the generalization of image captioning models. Constructing a large-scale labeled image captioning dataset is an expensive task in terms of labor, time, and cost. In contrast to manually annotating all the training samples, separately collecting uni-modal datasets is immensely easier, e.g., a large-scale image dataset and a sentence dataset. We leverage such massive unpaired image and caption data upon standard paired data by learning to associate them. To this end, our proposed semi-supervised learning method assigns pseudo-labels to unpaired samples in an adversarial learning fashion, where the joint distribution of image and caption is learned. Our method trains a captioner to learn from a paired data and to progressively associate unpaired data. This approach shows noticeable performance improvement even in challenging scenarios including out-of-task data (i.e., relational captioning, where the target task is different from the unpaired data) and web-crawled data. We also show that our proposed method is theoretically well-motivated and has a favorable global optimal property. Our extensive and comprehensive empirical results both on (1) image-based and (2) dense region-based captioning datasets followed by comprehensive analysis on the scarcely-paired COCO dataset demonstrate the consistent effectiveness of our semisupervised learning method with unpaired data compared to competing methods.

翻译：我们提出了一种新颖的数据高效半监督框架，以提升图像描述生成模型的泛化能力。构建大规模带标注的图像描述数据集在人力、时间和成本上均是一项昂贵的任务。与手动标注所有训练样本相比，分别收集单模态数据集（例如大规模图像数据集和句子数据集）要容易得多。我们通过学习关联这些数据，在标准配对数据的基础上利用大量无配对的图像与描述数据。为此，我们提出的半监督学习方法以对抗学习的方式为无配对样本分配伪标签，在此过程中学习图像与描述的联合分布。我们的方法训练描述生成器从配对数据中学习，并逐步关联无配对数据。该方法即使在具有挑战性的场景中（包括任务外数据——即关系型描述生成，其目标任务与无配对数据不同——以及网络爬取数据）也显示出显著的性能提升。我们还证明了所提方法具有扎实的理论基础，并具备良好的全局最优特性。在（1）基于图像和（2）基于密集区域的描述生成数据集上的广泛且全面的实证结果，以及随后对稀疏配对COCO数据集的综合分析，均一致表明本半监督学习方法在使用无配对数据时相较于竞争方法具有持续有效性。