Image-to-recipe retrieval is a challenging vision-to-language task of significant practical value. The main challenge of the task lies in the ultra-high redundancy in the long recipe and the large variation reflected in both food item combination and food item appearance. A de-facto idea to address this task is to learn a shared feature embedding space in which a food image is aligned better to its paired recipe than other recipes. However, such supervised global matching is prone to supervision collapse, i.e., only partial information that is necessary for distinguishing training pairs can be identified, while other information that is potentially useful in generalization could be lost. To mitigate such a problem, we propose a mask-augmentation-based local matching network (MALM), where an image-text matching module and a masked self-distillation module benefit each other mutually to learn generalizable cross-modality representations. On one hand, we perform local matching between the tokenized representations of image and text to locate fine-grained cross-modality correspondence explicitly. We involve representations of masked image patches in this process to alleviate overfitting resulting from local matching especially when some food items are underrepresented. On the other hand, predicting the hidden representations of the masked patches through self-distillation helps to learn general-purpose image representations that are expected to generalize better. And the multi-task nature of the model enables the representations of masked patches to be text-aware and thus facilitates the lost information reconstruction. Experimental results on Recipe1M dataset show our method can clearly outperform state-of-the-art (SOTA) methods. Our code will be available at https://github.com/MyFoodChoice/MALM_Mask_Augmentation_based_Local_Matching-_for-_Food_Recipe_Retrieval

翻译：图像到食谱检索是一项具有重要实际价值的视觉到语言任务。该任务的主要挑战在于长食谱中存在的超高冗余性，以及食物组合与食物外观所反映的巨大差异。解决此任务的常规思路是学习一个共享特征嵌入空间，使食物图像与其配对食谱的对齐程度优于其他食谱。然而，这种基于监督的全局匹配容易陷入监督崩溃，即仅能识别区分训练样本对所需的局部信息，而其他可能对泛化有用的信息则会丢失。为缓解该问题，我们提出一种基于掩码增强的局部匹配网络（MALM），其中图像-文本匹配模块与掩码自蒸馏模块相互促进，共同学习可泛化的跨模态表示。一方面，我们对图像和文本的词元化表示进行局部匹配，以显式定位细粒度的跨模态对应关系。在此过程中引入掩码图像块的表示，以缓解因局部匹配（尤其当某些食物项目表示不足时）导致的过拟合。另一方面，通过自蒸馏预测掩码图像块的隐藏表示，有助于学习通用图像表示，从而提升泛化能力。模型的多任务特性使掩码图像块的表示具备文本感知能力，进而促进缺失信息的重建。在Recipe1M数据集上的实验结果表明，我们的方法明显优于现有最优方法（SOTA）。相关代码将公开于 https://github.com/MyFoodChoice/MALM_Mask_Augmentation_based_Local_Matching-_for-_Food_Recipe_Retrieval