Latent representation alignment has become a foundational technique for constructing multimodal large language models (MLLM) by mapping embeddings from different modalities into a shared space, often aligned with the embedding space of large language models (LLMs) to enable effective cross-modal understanding. While preliminary protein-focused MLLMs have emerged, they have predominantly relied on heuristic approaches, lacking a fundamental understanding of optimal alignment practices across representations. In this study, we explore the alignment of multimodal representations between LLMs and Geometric Deep Models (GDMs) in the protein domain. We comprehensively evaluate three state-of-the-art LLMs (Gemma2-2B, LLaMa3.1-8B, and LLaMa3.1-70B) with four protein-specialized GDMs (GearNet, GVP, ScanNet, GAT). Our work examines alignment factors from both model and protein perspectives, identifying challenges in current alignment methodologies and proposing strategies to improve the alignment process. Our key findings reveal that GDMs incorporating both graph and 3D structural information align better with LLMs, larger LLMs demonstrate improved alignment capabilities, and protein rarity significantly impacts alignment performance. We also find that increasing GDM embedding dimensions, using two-layer projection heads, and fine-tuning LLMs on protein-specific data substantially enhance alignment quality. These strategies offer potential enhancements to the performance of protein-related multimodal models. Our code and data are available at https://github.com/Tizzzzy/LLM-GDM-alignment.

翻译：潜在表征对齐已成为构建多模态大语言模型（MLLM）的一项基础技术，其通过将来自不同模态的嵌入映射到一个共享空间（通常与大语言模型（LLM）的嵌入空间对齐）来实现有效的跨模态理解。尽管初步的蛋白质导向MLLM已经出现，但它们主要依赖于启发式方法，缺乏对跨表征最优对齐实践的根本性理解。在本研究中，我们探索了蛋白质领域中LLM与几何深度学习模型（GDM）之间的多模态表征对齐。我们全面评估了三种最先进的LLM（Gemma2-2B、LLaMa3.1-8B和LLaMa3.1-70B）与四种蛋白质专用GDM（GearNet、GVP、ScanNet、GAT）。我们的工作从模型和蛋白质两个角度考察了对齐因素，识别了当前对齐方法中的挑战，并提出了改进对齐过程的策略。我们的关键发现表明：同时包含图结构和3D结构信息的GDM与LLM对齐效果更好；更大的LLM展现出更强的对齐能力；蛋白质的稀有性显著影响对齐性能。我们还发现，增加GDM嵌入维度、使用双层投影头以及在蛋白质特定数据上微调LLM，都能显著提升对齐质量。这些策略为提升蛋白质相关多模态模型的性能提供了潜在的改进途径。我们的代码和数据可在 https://github.com/Tizzzzy/LLM-GDM-alignment 获取。