Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we developed SpatialSoundQA, a spatial sound-based question-answering dataset, offering a range of QA tasks that train BAT in various aspects of spatial sound perception and reasoning. The acoustic front end encoder of BAT is a novel spatial audio encoder named Spatial Audio Spectrogram Transformer, or Spatial-AST, which by itself achieves strong performance across sound event detection, spatial localization, and distance estimation. By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment. Our experiments demonstrate BAT's superior performance on both spatial sound perception and reasoning, showcasing the immense potential of LLMs in navigating and interpreting complex spatial audio environments.

翻译：空间声音推理是人类的基本技能，使我们能够基于声音导航和解读环境。本文提出BAT模型，通过将双耳声学场景分析模型的空间声音感知能力与大语言模型的自然语言推理能力相结合，复现了这一先天能力。为解决现有野外空间声音数据集缺失的问题，我们利用AudioSet和SoundSpaces 2.0合成双耳音频数据集，并构建了基于空间声音的问答数据集SpatialSoundQA。该数据集提供多种问答任务，从空间声音感知与推理的多个维度训练BAT。BAT的声学前端编码器采用新型空间音频编码器——空间音频频谱图Transformer（Spatial-AST），该编码器在声音事件检测、空间定位和距离估计任务中均展现出卓越性能。通过将Spatial-AST与LLaMA-2 7B模型集成，BAT突破了传统声音事件定位与检测任务的边界，实现了对环境中声音关系的推理。实验表明，BAT在空间声音感知与推理任务中均表现优异，充分彰显了大语言模型在复杂空间音频环境导航与解读中的巨大潜力。