Molecular knowledge resides within three different modalities of information sources: molecular structures, biomedical documents, and knowledge bases. Effective incorporation of molecular knowledge from these modalities holds paramount significance in facilitating biomedical research. However, existing multimodal molecular foundation models exhibit limitations in capturing intricate connections between molecular structures and texts, and more importantly, none of them attempt to leverage a wealth of molecular expertise derived from knowledge graphs. In this study, we introduce MolFM, a multimodal molecular foundation model designed to facilitate joint representation learning from molecular structures, biomedical texts, and knowledge graphs. We propose cross-modal attention between atoms of molecular structures, neighbors of molecule entities and semantically related texts to facilitate cross-modal comprehension. We provide theoretical analysis that our cross-modal pre-training captures local and global molecular knowledge by minimizing the distance in the feature space between different modalities of the same molecule, as well as molecules sharing similar structures or functions. MolFM achieves state-of-the-art performance on various downstream tasks. On cross-modal retrieval, MolFM outperforms existing models with 12.13% and 5.04% absolute gains under the zero-shot and fine-tuning settings, respectively. Furthermore, qualitative analysis showcases MolFM's implicit ability to provide grounding from molecular substructures and knowledge graphs. Code and models are available on https://github.com/BioFM/OpenBioMed.

翻译：分子知识存在于三种不同模态的信息源中：分子结构、生物医学文献和知识库。有效整合这些模态中的分子知识对于促进生物医学研究至关重要。然而，现有的大规模多模态分子基础模型在捕捉分子结构与文本之间的复杂联系方面存在局限性，更重要的是，它们均未尝试利用知识图谱中蕴含的丰富分子专业知识。在本研究中，我们提出MolFM，一种旨在实现分子结构、生物医学文本和知识图谱联合表示学习的多模态分子基础模型。我们设计了分子结构中的原子、分子实体的邻居以及语义相关文本之间的跨模态注意力机制，以促进跨模态理解。我们通过理论分析证明，我们的跨模态预训练通过最小化同一分子不同模态之间以及共享相似结构或功能的分子在特征空间中的距离，捕获了局部和全局的分子知识。MolFM在各种下游任务上取得了最先进的性能。在跨模态检索任务中，MolFM在零样本和微调设置下分别以12.13%和5.04%的绝对提升优于现有模型。此外，定性分析展示了MolFM隐含的从分子子结构和知识图谱中提供依据的能力。代码和模型可在https://github.com/BioFM/OpenBioMed获取。