The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M$^{2}$UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M$^{2}$UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M$^{2}$UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.

翻译：当前，利用大语言模型（LLMs）的研究领域正经历蓬勃发展。许多工作借助这些模型的强大推理能力来理解多种模态（如文本、语音、图像、视频等），并运用LLMs理解人类意图以生成所需的输出（如图像、视频和音乐）。然而，将理解与生成能力相结合的LLMs研究仍处于探索阶段且成果有限。为解决这一局限，我们提出多模态音乐理解与生成（M$^{2}$UGen）框架，该框架融合了LLMs对音乐的理解与生成能力，可跨模态处理音乐内容。M$^{2}$UGen框架专为释放不同灵感来源的创作潜力而设计，通过分别采用预训练的MERT、ViT和ViViT模型，支持音乐、图像和视频等多模态输入。在音乐生成方面，我们探索了AudioLDM 2和MusicGen的应用。通过集成LLaMA 2模型，实现了多模态理解与音乐生成的桥梁式衔接。此外，我们利用MU-LLaMA模型生成大规模数据集，支持文本/图像/视频到音乐的生成任务，从而助力M$^{2}$UGen框架的训练。我们对所提框架进行了全面评估，实验结果表明，我们的模型已达到或超越当前最优模型的性能水平。