Automated Audio Captioning aims to describe the semantic content of input audio. Recent works have employed large language models (LLMs) as a text decoder to leverage their reasoning capabilities. However, prior approaches that project audio features into the LLM embedding space without considering cross-modal alignment fail to fully utilize these capabilities. To address this, we propose LAMB, an LLM-based audio captioning framework that bridges the modality gap between audio embeddings and the LLM text embedding space. LAMB incorporates a Cross-Modal Aligner that minimizes Cauchy-Schwarz divergence while maximizing mutual information, yielding tighter alignment between audio and text at both global and token levels. We further design a Two-Stream Adapter that extracts semantically enriched audio embeddings, thereby delivering richer information to the Cross-Modal Aligner. Finally, leveraging the aligned audio embeddings, a proposed Token Guide directly computes scores within the LLM text embedding space to steer the output logits of generated captions. Experimental results confirm that our framework strengthens the reasoning capabilities of the LLM decoder, achieving state-of-the-art performance on AudioCaps.

翻译：自动音频描述旨在描述输入音频的语义内容。近期研究采用大语言模型作为文本解码器，以利用其推理能力。然而，现有方法在将音频特征投影到LLM嵌入空间时未考虑跨模态对齐，导致无法充分发挥这些能力。为此，我们提出LAMB——一种基于LLM的音频描述框架，旨在桥接音频嵌入与LLM文本嵌入空间之间的模态鸿沟。LAMB引入跨模态对齐器，通过最小化柯西-施瓦茨散度并最大化互信息，实现音频与文本在全局和词元层面的紧密对齐。我们进一步设计双流适配器，提取语义增强的音频嵌入，从而为跨模态对齐器提供更丰富的信息。最后，利用对齐后的音频嵌入，所提出的词元引导器直接在LLM文本嵌入空间内计算得分，以引导生成描述的输出逻辑。实验结果证实，我们的框架增强了LLM解码器的推理能力，在AudioCaps数据集上实现了最先进的性能。