Automatic target sound extraction (TSE) is a machine learning approach to mimic the human auditory perception capability of attending to a sound source of interest from a mixture of sources. It often uses a model conditioned on a fixed form of target sound clues, such as a sound class label, which limits the ways in which users can interact with the model to specify the target sounds. To leverage variable number of clues cross modalities available in the inference phase, including a video, a sound event class, and a text caption, we propose a unified transformer-based TSE model architecture, where a multi-clue attention module integrates all the clues across the modalities. Since there is no off-the-shelf benchmark to evaluate our proposed approach, we build a dataset based on public corpora, Audioset and AudioCaps. Experimental results for seen and unseen target-sound evaluation sets show that our proposed TSE model can effectively deal with a varying number of clues which improves the TSE performance and robustness against partially compromised clues.

翻译：自动目标声音提取（TSE）是一种机器学习方法，旨在模仿人类从混合声源中关注特定声源的听觉感知能力。该方法通常使用基于固定形式目标声音线索（如声音类别标签）的模型，这限制了用户通过交互指定目标声音的方式。为利用推理阶段可用的跨模态可变数量线索（包括视频、声音事件类别和文本描述），我们提出了一种统一的基于Transformer的TSE模型架构，其中多线索注意力模块集成了跨模态的所有线索。由于缺乏现成基准来评估所提方法，我们基于公开语料库Audioset和AudioCaps构建了数据集。在可见与不可见目标声音评估集上的实验结果表明，所提TSE模型能够有效处理可变数量的线索，从而提升TSE性能及对部分受损线索的鲁棒性。