Artificial neural networks typically struggle in generalizing to out-of-context examples. One reason for this limitation is caused by having datasets that incorporate only partial information regarding the potential correlational structure of the world. In this work, we propose TIDA (Targeted Image-editing Data Augmentation), a targeted data augmentation method focused on improving models' human-like abilities (e.g., gender recognition) by filling the correlational structure gap using a text-to-image generative model. More specifically, TIDA identifies specific skills in captions describing images (e.g., the presence of a specific gender in the image), changes the caption (e.g., "woman" to "man"), and then uses a text-to-image model to edit the image in order to match the novel caption (e.g., uniquely changing a woman to a man while maintaining the context identical). Based on the Flickr30K benchmark, we show that, compared with the original data set, a TIDA-enhanced dataset related to gender, color, and counting abilities induces better performance in several image captioning metrics. Furthermore, on top of relying on the classical BLEU metric, we conduct a fine-grained analysis of the improvements of our models against the baseline in different ways. We compared text-to-image generative models and found different behaviors of the image captioning models in terms of encoding visual encoding and textual decoding.

翻译：摘要：人工神经网络通常难以泛化至脱离常规语境（out-of-context）的样本。这一局限的部分原因在于，现有数据集仅包含世界潜在关联结构的片面信息。本研究提出定向图像编辑数据增强方法（TIDA），通过利用文本到图像生成模型填补关联结构空白，专用于提升模型的类人能力（如性别识别）。具体而言，TIDA首先识别描述图像的文本中的特定技能（如图像中是否存在特定性别），修改文本（如将"女性"改为"男性"），随后使用文本到图像模型编辑图像以匹配新文本（例如，在保持上下文完全一致的前提下，仅将女性改为男性）。基于Flickr30K基准的实验表明：相较于原始数据集，经TIDA增强的性别、颜色及计数能力相关数据集，在多项图像描述评价指标上显著提升模型性能。此外，除依赖传统BLEU指标外，我们通过细粒度分析，从多个维度探究模型相较于基线方法的改进。通过对比不同文本到图像生成模型，我们发现了图像描述模型在视觉编码与文本解码过程中的差异化表现。