The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M$^{2}$UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M$^{2}$UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M$^{2}$UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.

翻译：当前利用大语言模型的研究正呈现蓬勃发展之势。众多研究工作借助这些模型强大的推理能力来理解文本、语音、图像、视频等多种模态信息，同时利用大语言模型理解人类意图并生成图像、视频、音乐等所需内容。然而，结合理解与生成两大功能的大语言模型研究仍属少数且尚处起步阶段。为填补这一空白，我们提出多模态音乐理解与生成（M$^{2}$UGen）框架，该框架整合了大语言模型的能力，可针对不同模态实现音乐的"理解"与"生成"功能。M$^{2}$UGen框架专为挖掘多元灵感来源的创作潜力而设计，通过分别采用预训练的MERT、ViT和ViViT模型，能够处理音乐、图像和视频三种模态信息。在音乐生成方面，我们探索了AudioLDM 2和MusicGen的应用。通过集成LLaMA 2模型，实现了多模态理解与音乐生成之间的桥梁搭建。此外，我们利用MU-LLaMA模型生成支持文本/图像/视频到音乐生成的大规模数据集，为M$^{2}$UGen框架的训练提供支撑。我们对所提框架进行了全面评估，实验结果表明该模型已达到或超越当前最先进模型的性能水平。