Traditional recommender systems heavily rely on ID features, which often encounter challenges related to cold-start and generalization. Modeling pre-extracted content features can mitigate these issues, but is still a suboptimal solution due to the discrepancies between training tasks and model parameters. End-to-end training presents a promising solution for these problems, yet most of the existing works mainly focus on retrieval models, leaving the multimodal techniques under-utilized. In this paper, we propose an industrial multimodal recommendation framework named EM3: End-to-end training of Multimodal Model and ranking Model, which sufficiently utilizes multimodal information and allows personalized ranking tasks to directly train the core modules in the multimodal model to obtain more task-oriented content features, without overburdening resource consumption. First, we propose Fusion-Q-Former, which consists of transformers and a set of trainable queries, to fuse different modalities and generate fixed-length and robust multimodal embeddings. Second, in our sequential modeling for user content interest, we utilize Low-Rank Adaptation technique to alleviate the conflict between huge resource consumption and long sequence length. Third, we propose a novel Content-ID-Contrastive learning task to complement the advantages of content and ID by aligning them with each other, obtaining more task-oriented content embeddings and more generalized ID embeddings. In experiments, we implement EM3 on different ranking models in two scenario, achieving significant improvements in both offline evaluation and online A/B test, verifying the generalizability of our method. Ablation studies and visualization are also performed. Furthermore, we also conduct experiments on two public datasets to show that our proposed method outperforms the state-of-the-art methods.

翻译：传统推荐系统严重依赖ID特征，常面临冷启动与泛化性挑战。对预提取内容特征进行建模可缓解这些问题，但由于训练任务与模型参数之间的差异，该方法仍非最优解。端到端训练为上述问题提供了有前景的解决方案，但现有工作主要聚焦检索模型，未能充分利用多模态技术。本文提出工业级多模态推荐框架EM3（多模态模型与排序模型的端到端训练），该框架充分挖掘多模态信息，允许个性化排序任务直接训练多模态模型中的核心模块以获取更具任务导向性的内容特征，且不会过度增加资源消耗。首先，我们提出由Transformer和可训练查询集构成的Fusion-Q-Former，用于融合不同模态并生成固定长度的鲁棒多模态嵌入。其次，在用户内容兴趣序列建模中，我们采用低秩自适应技术缓解资源消耗与长序列长度之间的矛盾。第三，我们提出新型内容-ID对比学习任务，通过将内容与ID特征相互对齐来互补各自优势，从而获得更任务导向的内容嵌入与更具泛化性的ID嵌入。实验环节，我们在两个场景的不同排序模型上部署EM3，离线评估与在线A/B测试均取得显著提升，验证了方法的泛化能力。此外还进行了消融实验与可视化分析。我们在两个公开数据集上的实验表明，所提方法优于现有最先进方法。