In the context of environmental sound classification, the adaptability of systems is key: which sound classes are interesting depends on the context and the user's needs. Recent advances in text-to-audio retrieval allow for zero-shot audio classification, but performance compared to supervised models remains limited. This work proposes a multimodal prototypical approach that exploits local audio-text embeddings to provide more relevant answers to audio queries, augmenting the adaptability of sound detection in the wild. We do this by first using text to query a nearby community of audio embeddings that best characterize each query sound, and select the group's centroids as our prototypes. Second, we compare unseen audio to these prototypes for classification. We perform multiple ablation studies to understand the impact of the embedding models and prompts. Our unsupervised approach improves upon the zero-shot state-of-the-art in three sound recognition benchmarks by an average of 12%.

翻译：在环境声音分类的背景下，系统的适应性至关重要：哪些声音类别值得关注取决于具体语境和用户需求。文本到音频检索的最新进展实现了零样本音频分类，但与监督模型相比，其性能仍然有限。本文提出一种多模态原型方法，利用局部音频-文本嵌入为音频查询提供更相关的答案，从而增强野外声音检测的适应性。具体而言，我们首先使用文本查询邻近的音频嵌入社区，以最佳表征每个查询声音，并将该社区的质心选作原型。其次，我们将未见过的音频与这些原型进行比较以进行分类。我们进行了多项消融研究，以理解嵌入模型和提示词的影响。我们的无监督方法在三个声音识别基准测试中，相较于现有零样本最佳方法平均提升了12%。