Learning recipe and food image representation in common embedding space is non-trivial but crucial for cross-modal recipe retrieval. In this paper, we propose CAR framework with three novel techniques, i.e., Consolidation, Augmentation and Regulation, for cross-modal recipe retrieval. We introduce adapter layers to consolidate pre-trained CLIP model with much less computation cost than fully cumbersome fine-tuning all the parameters. Furthermore, leveraging on the strong capability of foundation models (i.e., SAM and LLM), we propose to augment recipe and food image by extracting information related to the counterpart. SAM generates image segments corresponding to ingredients in the recipe, while LLM produces a visual imagination description from the recipe, aiming to capture the visual cues of a food image. In addition, we introduce circle loss to regulate cross-modal embedding space, which assigns different penalties for positive and negative pairs. With the extra augmented data from recipe and image, multi-level circle loss is proposed, which applies circle loss not only to original image-recipe pairs, but also to image segments and recipe, visual imagination description and food image as well as any two sections within a recipe. On Recipe1M dataset, our proposed CAR outperforms all the existing methods by a large margin. Extensive ablation studies are conducted to validate the effectiveness of each component of CAR. We will make our code and models publicly available.

翻译：在公共嵌入空间中学习食谱与食物图像表征是实现跨模态食谱检索的关键挑战。本文提出CAR框架，包含三项创新技术：整合、增强与调控。通过引入适配层整合预训练CLIP模型，相较于完全微调所有参数，该方法大幅降低计算成本。进一步地，我们利用基础模型（SAM与LLM）的强表征能力，通过提取与对应模态相关的信息来增强食谱与食物图像：SAM生成与食谱中食材对应的图像分割区域，LLM则基于食谱生成视觉想象描述以捕捉食物图像的视觉线索。同时，我们引入圆损失函数调控跨模态嵌入空间，对正负样本对施加差异化惩罚。结合食谱与图像的增强数据，提出多级圆损失——该损失不仅作用于原始图像-食谱对，还扩展至图像分割区域与食谱、视觉想象描述与食物图像、以及食谱内部任意两段文本之间的匹配。在Recipe1M数据集上，我们的CAR方法以显著优势超越现有全部方法。通过广泛消融实验验证了CAR各组件的有效性，代码与模型将公开发布。