The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M$^{2}$UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M$^{2}$UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M$^{2}$UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.

翻译：当前利用大型语言模型（LLMs）的研究领域正蓬勃发展。许多工作借助这些模型强大的推理能力来理解多种模态，如文本、语音、图像、视频等，同时利用LLMs理解人类意图并生成所需的输出，如图像、视频和音乐。然而，结合理解与生成能力的LLMs研究仍较为有限且处于起步阶段。为填补这一空白，我们提出了一种多模态音乐理解与生成（M$^{2}$UGen）框架，该框架整合了LLM的能力，以理解和生成不同模态下的音乐。M$^{2}$UGen框架专为从多样灵感源中释放创造力而设计，通过分别使用预训练的MERT、ViT和ViViT模型处理音乐、图像和视频，实现跨模态理解。为实现音乐生成，我们探索了AudioLDM 2和MusicGen的使用。通过集成LLaMA 2模型，多模态理解与音乐生成得以桥接。此外，我们利用MU-LLaMA模型生成支持文本/图像/视频到音乐生成的大规模数据集，从而促进M$^{2}$UGen框架的训练。我们对所提出的框架进行了全面评估。实验结果表明，我们的模型达到或超越了当前最先进模型的性能。