We introduce C3LLM (Conditioned-on-Three-Modalities Large Language Models), a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities, synthesizing the given conditional information, and making multimodal generation in a discrete manner. Our contributions are as follows. First, we adapt a hierarchical structure for audio generation tasks with pre-trained audio codebooks. Specifically, we train the LLM to generate audio semantic tokens from the given conditions, and further use a non-autoregressive transformer to generate different levels of acoustic tokens in layers to better enhance the fidelity of the generated audio. Second, based on the intuition that LLMs were originally designed for discrete tasks with the next-word prediction method, we use the discrete representation for audio generation and compress their semantic meanings into acoustic tokens, similar to adding "acoustic vocabulary" to LLM. Third, our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model, providing more versatility in an end-to-end fashion. Our C3LLM achieves improved results through various automated evaluation metrics, providing better semantic alignment compared to previous methods.

翻译：本文提出C3LLM（三模态条件大语言模型），这是一种将视频到音频、音频到文本以及文本到音频三项任务相结合的新型框架。C3LLM采用大语言模型（LLM）结构作为桥梁，用于对齐不同模态、合成给定条件信息，并以离散方式实现多模态生成。我们的贡献如下：首先，我们采用预训练音频码本构建分层结构以处理音频生成任务。具体而言，我们训练LLM根据给定条件生成音频语义标记，并进一步使用非自回归Transformer分层生成不同层级的声学标记，从而提升生成音频的保真度。其次，基于LLM最初设计用于通过下一词预测方法处理离散任务的特性，我们采用离散表示进行音频生成，并将其语义信息压缩为声学标记，这类似于为LLM添加“声学词汇表”。第三，我们的方法将音频理解、视频到音频生成以及文本到音频生成等先前任务整合至统一模型中，以端到端方式提供更强的多功能性。实验表明，C3LLM在多项自动评估指标上取得改进效果，与现有方法相比实现了更好的语义对齐。