The online emergence of multi-modal sharing platforms (eg, TikTok, Youtube) is powering personalized recommender systems to incorporate various modalities (eg, visual, textual and acoustic) into the latent user representations. While existing works on multi-modal recommendation exploit multimedia content features in enhancing item embeddings, their model representation capability is limited by heavy label reliance and weak robustness on sparse user behavior data. Inspired by the recent progress of self-supervised learning in alleviating label scarcity issue, we explore deriving self-supervision signals with effectively learning of modality-aware user preference and cross-modal dependencies. To this end, we propose a new Multi-Modal Self-Supervised Learning (MMSSL) method which tackles two key challenges. Specifically, to characterize the inter-dependency between the user-item collaborative view and item multi-modal semantic view, we design a modality-aware interactive structure learning paradigm via adversarial perturbations for data augmentation. In addition, to capture the effects that user's modality-aware interaction pattern would interweave with each other, a cross-modal contrastive learning approach is introduced to jointly preserve the inter-modal semantic commonality and user preference diversity. Experiments on real-world datasets verify the superiority of our method in offering great potential for multimedia recommendation over various state-of-the-art baselines. The implementation is released at: https://github.com/HKUDS/MMSSL.

翻译：多模态共享平台（如抖音、YouTube）的在线兴起，正推动个性化推荐系统将多种模态（如视觉、文本和音频）融入用户隐式表示。现有基于多模态推荐的工作虽利用多媒体内容特征增强物品嵌入，但其模型表示能力受限于对标签的强依赖以及在稀疏用户行为数据上的弱鲁棒性。受自监督学习在缓解标签稀缺问题上的最新进展启发，我们探索在有效学习模态感知的用户偏好及跨模态依赖过程中提取自监督信号。为此，我们提出一种新型多模态自监督学习方法（MMSSL），旨在解决两大关键挑战。具体而言：为刻画用户-物品协同视图与物品多模态语义视图间的相互依赖关系，我们设计了一种通过对抗扰动实现数据增强的模态感知交互结构学习范式；此外，为捕获用户模态感知交互模式相互交织的影响，我们引入跨模态对比学习方法，以联合保持模态间语义共性与用户偏好多样性。在真实数据集上的实验证明，该方法在多媒体推荐中优于多种当前最优基线方法，展现出巨大潜力。实现代码已开源在：https://github.com/HKUDS/MMSSL。