Speech classification tasks often require powerful language understanding models to grasp useful features, which becomes problematic when limited training data is available. To attain superior classification performance, we propose to harness the inherent value of multimodal representations by transcribing speech using automatic speech recognition (ASR) models and translating the transcripts into different languages via pretrained translation models. We thus obtain an audio-textual (multimodal) representation for each data sample. Subsequently, we combine language-specific Bidirectional Encoder Representations from Transformers (BERT) with Wav2Vec2.0 audio features via a novel cascaded cross-modal transformer (CCMT). Our model is based on two cascaded transformer blocks. The first one combines text-specific features from distinct languages, while the second one combines acoustic features with multilingual features previously learned by the first transformer block. We employed our system in the Requests Sub-Challenge of the ACM Multimedia 2023 Computational Paralinguistics Challenge. CCMT was declared the winning solution, obtaining an unweighted average recall (UAR) of 65.41% and 85.87% for complaint and request detection, respectively. Moreover, we applied our framework on the Speech Commands v2 and HarperValleyBank dialog data sets, surpassing previous studies reporting results on these benchmarks. Our code is freely available for download at: https://github.com/ristea/ccmt.

翻译：语音分类任务通常需要强大的语言理解模型来提取有用特征，这在训练数据有限时存在困难。为获得更优的分类性能，我们提出利用多模态表示的内在价值——通过自动语音识别（ASR）模型转录语音，并利用预训练翻译模型将转录文本翻译成不同语言。由此为每个数据样本获取音频-文本（多模态）表示。随后，我们通过新型级联交叉模态变换器（CCMT），将语言特定的双向编码器表示（BERT）与Wav2Vec2.0音频特征相结合。该模型基于两个级联的Transformer模块：第一个模块融合来自不同语言的文本特定特征，第二个模块将声学特征与第一个Transformer模块先前学习的多语言特征相结合。我们在ACM Multimedia 2023计算副语言学挑战赛的请求子挑战中应用了该系统。CCMT被评为优胜方案，在投诉检测和请求检测上分别获得65.41%和85.87%的未加权平均召回率（UAR）。此外，我们在Speech Commands v2和HarperValleyBank对话数据集上应用该框架，超越了此前在这些基准上报告的研究成果。我们的代码已在https://github.com/ristea/ccmt开源下载。