While embeddings from multimodal large language models (LLMs) excel as general-purpose representations, their application to dynamic modalities like audio and video remains underexplored. We introduce WAVE (\textbf{u}nified \& \textbf{v}ersatile \textbf{a}udio-\textbf{v}isual \textbf{e}mbeddings), the first LLM-based embedding that creates a unified representation space for text, audio, and video modalities. WAVE employs a novel hierarchical feature fusion strategy and a joint multi-modal, multi-task training approach to enable two key capabilities: any-to-any cross-modal retrieval and the generation of prompt-aware embeddings tailored to user instructions. Experimentally, WAVE sets a new state-of-the-art on the MMEB-v2 video benchmark and achieves superior results in audio and video-to-audio retrieval. Its prompt-aware nature also yields remarkable performance in multimodal question answering, significantly outperforming existing embedding models. Ablation studies validate our joint training strategy, demonstrating improved performance across all modalities. With a newly introduced benchmark for versatile audio-visual learning, WAVE opens up broad possibilities for cross-modal, any-to-any applications. Our code and checkpoints are released at \href{https://github.com/TCL606/WAVE}{https://github.com/TCL606/WAVE}.

翻译：尽管多模态大语言模型（LLM）生成的嵌入作为通用表征表现出色，但其在音频和视频等动态模态中的应用仍待深入探索。我们提出了WAVE（统一且通用的音视频嵌入），这是首个基于LLM的嵌入模型，能够为文本、音频和视频模态创建统一的表征空间。WAVE采用新颖的层次化特征融合策略与联合多模态多任务训练方法，实现了两大核心能力：任意模态间的跨模态检索，以及根据用户指令生成具有提示感知能力的嵌入。实验表明，WAVE在MMEB-v2视频基准测试中取得了最先进的性能，并在音频检索及视频到音频检索任务中表现优异。其提示感知特性还在多模态问答任务中展现出卓越性能，显著超越了现有嵌入模型。消融研究验证了联合训练策略的有效性，证明其能全面提升各模态性能。通过新构建的通用音视频学习基准，WAVE为跨模态任意到任意应用开辟了广阔前景。我们的代码与模型检查点已发布于\href{https://github.com/TCL606/WAVE}{https://github.com/TCL606/WAVE}。