The online emergence of multi-modal sharing platforms (eg, TikTok, Youtube) is powering personalized recommender systems to incorporate various modalities (eg, visual, textual and acoustic) into the latent user representations. While existing works on multi-modal recommendation exploit multimedia content features in enhancing item embeddings, their model representation capability is limited by heavy label reliance and weak robustness on sparse user behavior data. Inspired by the recent progress of self-supervised learning in alleviating label scarcity issue, we explore deriving self-supervision signals with effectively learning of modality-aware user preference and cross-modal dependencies. To this end, we propose a new Multi-Modal Self-Supervised Learning (MMSSL) method which tackles two key challenges. Specifically, to characterize the inter-dependency between the user-item collaborative view and item multi-modal semantic view, we design a modality-aware interactive structure learning paradigm via adversarial perturbations for data augmentation. In addition, to capture the effects that user's modality-aware interaction pattern would interweave with each other, a cross-modal contrastive learning approach is introduced to jointly preserve the inter-modal semantic commonality and user preference diversity. Experiments on real-world datasets verify the superiority of our method in offering great potential for multimedia recommendation over various state-of-the-art baselines. The implementation is released at: https://github.com/HKUDS/MMSSL.

翻译：多模态共享平台（如抖音、YouTube）的线上涌现，推动了个性化推荐系统将多种模态（如视觉、文本和音频）融入潜在用户表征。现有关于多模态推荐的研究虽利用多媒体内容特征增强物品嵌入，但其模型表征能力受制于对标签的严重依赖以及稀疏用户行为数据上的弱鲁棒性。受自监督学习在缓解标签稀缺问题方面最新进展的启发，我们探索通过有效学习模态感知的用户偏好与跨模态依赖关系来提取自监督信号。为此，我们提出了一种新的多模态自监督学习方法（MMSSL），以应对两个关键挑战。具体而言，为刻画用户-物品协同视图与物品多模态语义视图间的相互依赖关系，我们设计了一种基于对抗扰动的模态感知交互式结构学习范式进行数据增强。此外，为捕捉用户模态感知交互模式相互交织的影响，我们引入了一种跨模态对比学习方法，以同时保留模态间语义共性与用户偏好多样性。在真实数据集上的实验验证了我们的方法相较于各类最先进基线在多媒体推荐中的显著优势。相关实现已开源于：https://github.com/HKUDS/MMSSL。