Deep neural networks have been applied to audio spectrograms for respiratory sound classification. Existing models often treat the spectrogram as a synthetic image while overlooking its physical characteristics. In this paper, a Multi-View Spectrogram Transformer (MVST) is proposed to embed different views of time-frequency characteristics into the vision transformer. Specifically, the proposed MVST splits the mel-spectrogram into different sized patches, representing the multi-view acoustic elements of a respiratory sound. These patches and positional embeddings are then fed into transformer encoders to extract the attentional information among patches through a self-attention mechanism. Finally, a gated fusion scheme is designed to automatically weigh the multi-view features to highlight the best one in a specific scenario. Experimental results on the ICBHI dataset demonstrate that the proposed MVST significantly outperforms state-of-the-art methods for classifying respiratory sounds.

翻译：深度神经网络已被应用于音频频谱图以进行呼吸音分类。现有模型往往将频谱图视为合成图像，而忽略其物理特性。本文提出一种多视角频谱图变压器（MVST），将时频特性的不同视角嵌入到视觉变压器中。具体而言，所提出的MVST将梅尔频谱图分割为不同尺寸的面片，代表呼吸音的多视角声学元素。这些面片与位置编码随后被送入变压器编码器，通过自注意力机制提取面片间的注意力信息。最后，设计一种门控融合方案来自动加权多视角特征，以突出特定场景下的最优特征。在ICBHI数据集上的实验结果表明，所提出的MVST在呼吸音分类任务上显著优于现有最优方法。