The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M$^{2}$UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M$^{2}$UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M$^{2}$UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.

翻译：当前利用大语言模型（LLMs）的研究领域正蓬勃发展。众多研究借助这些模型强大的推理能力来理解文本、语音、图像、视频等多模态信息，并利用LLMs理解人类意图，生成图像、视频、音乐等预期输出。然而，融合理解与生成能力的LLM研究仍较为有限，尚处于早期探索阶段。为弥补这一空白，我们提出多模态音乐理解与生成（M$^{2}$UGen）框架，该框架整合LLM能力，实现对不同模态音乐的理解与生成。M$^{2}$UGen框架专为从多样灵感源（包括音乐、图像和视频）中释放创造力而设计，分别采用预训练的MERT、ViT和ViViT模型进行处理。在音乐生成方面，我们探索了AudioLDM 2与MusicGen的应用。通过集成LLaMA 2模型，实现了多模态理解与音乐生成的桥接。此外，我们利用MU-LLaMA模型生成大规模数据集以支持文本/图像/视频到音乐的生成任务，从而助力M$^{2}$UGen框架的训练。我们对所提出的框架进行了全面评估，实验结果表明，该模型已达到或超越现有最先进模型的性能水平。