Cross-modal retrieval between food images and recipe texts is an important task with applications in nutritional management, dietary logging, and cooking assistance. Existing methods predominantly rely on dual-encoder architectures with separate image and text encoders, requiring complex alignment strategies and task-specific network designs to bridge the semantic gap between modalities. In this work, we propose SIMMER (Single Integrated Multimodal Model for Embedding Recipes), which applies Multimodal Large Language Model (MLLM)-based embedding models, specifically VLM2Vec, to this task, replacing the conventional dual-encoder paradigm with a single unified encoder that processes both food images and recipe texts. We design prompt templates tailored to the structured nature of recipes, which consist of a title, ingredients, and cooking instructions, enabling effective embedding generation by the MLLM. We further introduce a component-aware data augmentation strategy that trains the model on both complete and partial recipes, improving robustness to incomplete inputs. Experiments on the Recipe1M dataset demonstrate that SIMMER achieves state-of-the-art performance across both the 1k and 10k evaluation settings, substantially outperforming all prior methods. In particular, our best model improves the 1k image-to-recipe R@1 from 81.8\% to 87.5\% and the 10k image-to-recipe R@1 from 56.5\% to 65.5\% compared to the previous best method.

翻译：跨模态食品图像与食谱文本之间的检索是一项重要任务，在营养管理、饮食记录和烹饪辅助等领域具有应用价值。现有方法主要依赖双编码器架构，通过分离的图像和文本编码器，需要复杂的对齐策略和特定任务网络设计来弥合模态间的语义鸿沟。本文提出SIMMER（面向食谱嵌入的单集成多模态模型），将基于多模态大语言模型（MLLM）的嵌入模型（尤其是VLM2Vec）应用于此任务，用统一编码器替代传统双编码器范式，该编码器可同时处理食品图像和食谱文本。针对由标题、配料和烹饪步骤构成的食谱结构化特性，我们设计了专属提示模板，使MLLM能够高效生成嵌入。进一步提出组件感知的数据增强策略，训练模型同时处理完整和部分食谱，提升对不完整输入的鲁棒性。在Recipe1M数据集上的实验表明，SIMMER在1k和10k评估设置下均达到最优性能，显著优于所有现有方法。值得注意的是，与先前最佳方法相比，我们的最优模型将1k图像到食谱的R@1从81.8%提升至87.5%，将10k图像到食谱的R@1从56.5%提升至65.5%。