We propose a method to recommend music for an input video while allowing a user to guide music selection with free-form natural language. A key challenge of this problem setting is that existing music video datasets provide the needed (video, music) training pairs, but lack text descriptions of the music. This work addresses this challenge with the following three contributions. First, we propose a text-synthesis approach that relies on an analogy-based prompting procedure to generate natural language music descriptions from a large-scale language model (BLOOM-176B) given pre-trained music tagger outputs and a small number of human text descriptions. Second, we use these synthesized music descriptions to train a new trimodal model, which fuses text and video input representations to query music samples. For training, we introduce a text dropout regularization mechanism which we show is critical to model performance. Our model design allows for the retrieved music audio to agree with the two input modalities by matching visual style depicted in the video and musical genre, mood, or instrumentation described in the natural language query. Third, to evaluate our approach, we collect a testing dataset for our problem by annotating a subset of 4k clips from the YT8M-MusicVideo dataset with natural language music descriptions which we make publicly available. We show that our approach can match or exceed the performance of prior methods on video-to-music retrieval while significantly improving retrieval accuracy when using text guidance.

翻译：本文提出一种方法，在为输入视频推荐音乐的同时，允许用户通过自由形式的自然语言引导音乐选择。该问题设定的核心挑战在于：现有音乐视频数据集虽提供所需的（视频、音乐）训练对，但缺乏音乐的自然语言描述。本研究通过以下三方面贡献解决这一挑战。首先，我们提出一种文本合成方法，该方法基于类比提示流程，在给定预训练音乐标签器输出和少量人工文本描述的条件下，利用大规模语言模型（BLOOM-176B）生成自然语言音乐描述。其次，我们使用这些合成的音乐描述训练一种新的三模态模型，该模型融合文本与视频输入表示来查询音乐样本。在训练过程中，我们引入文本丢弃正则化机制，并证明该机制对模型性能至关重要。我们的模型设计通过匹配视频中呈现的视觉风格以及自然语言查询中描述的音乐流派、情绪或配器，使检索到的音乐音频与两种输入模态保持一致。第三，为评估方法，我们通过为YT8M-MusicVideo数据集中4000个片段子集标注自然语言音乐描述，构建了面向该问题的测试数据集，并公开发布。实验表明，我们的方法在视频到音乐检索任务上可达到或超越先前方法的性能，同时在使用文本引导时显著提升了检索准确率。