The online emergence of multi-modal sharing platforms (eg, TikTok, Youtube) is powering personalized recommender systems to incorporate various modalities (eg, visual, textual and acoustic) into the latent user representations. While existing works on multi-modal recommendation exploit multimedia content features in enhancing item embeddings, their model representation capability is limited by heavy label reliance and weak robustness on sparse user behavior data. Inspired by the recent progress of self-supervised learning in alleviating label scarcity issue, we explore deriving self-supervision signals with effectively learning of modality-aware user preference and cross-modal dependencies. To this end, we propose a new Multi-Modal Self-Supervised Learning (MMSSL) method which tackles two key challenges. Specifically, to characterize the inter-dependency between the user-item collaborative view and item multi-modal semantic view, we design a modality-aware interactive structure learning paradigm via adversarial perturbations for data augmentation. In addition, to capture the effects that user's modality-aware interaction pattern would interweave with each other, a cross-modal contrastive learning approach is introduced to jointly preserve the inter-modal semantic commonality and user preference diversity. Experiments on real-world datasets verify the superiority of our method in offering great potential for multimedia recommendation over various state-of-the-art baselines. The implementation is released at: https://github.com/HKUDS/MMSSL.

翻译：多模态分享平台（如抖音、YouTube）的在线兴起，推动个性化推荐系统将多种模态（如视觉、文本和声学）融入潜在用户表征。尽管现有关于多模态推荐的研究利用多媒体内容特征增强项目嵌入，其模型表征能力受限于对标签的强依赖性和稀疏用户行为数据上的弱鲁棒性。受自监督学习在缓解标签稀缺问题方面最新进展的启发，我们探索通过有效学习模态感知的用户偏好和跨模态依赖关系来导出自监督信号。为此，我们提出一种新的多模态自监督学习方法（MMSSL），其应对两个关键挑战。具体而言，为刻画用户-项目协同视图与项目多模态语义视图间的相互依赖关系，我们设计了一种通过对抗扰动进行数据增强的模态感知交互式结构学习范式。此外，为捕捉用户模态感知交互模式相互交织的影响，引入跨模态对比学习方法，共同保持模态间语义共性和用户偏好多样性。在真实数据集上的实验验证了该方法在多媒体推荐方面优于多种最新基线方法的显著潜力。代码已开源：https://github.com/HKUDS/MMSSL。