The lack of labeled data is a common challenge in speech classification tasks, particularly those requiring extensive subjective assessment, such as cognitive state classification. In this work, we propose a Semi-Supervised Learning (SSL) framework, introducing a novel multi-view pseudo-labeling method that leverages both acoustic and linguistic characteristics to select the most confident data for training the classification model. Acoustically, unlabeled data are compared to labeled data using the Frechet audio distance, calculated from embeddings generated by multiple audio encoders. Linguistically, large language models are prompted to revise automatic speech recognition transcriptions and predict labels based on our proposed task-specific knowledge. High-confidence data are identified when pseudo-labels from both sources align, while mismatches are treated as low-confidence data. A bimodal classifier is then trained to iteratively label the low-confidence data until a predefined criterion is met. We evaluate our SSL framework on emotion recognition and dementia detection tasks. Experimental results demonstrate that our method achieves competitive performance compared to fully supervised learning using only 30% of the labeled data and significantly outperforms two selected baselines.

翻译：在语音分类任务中，标注数据的缺乏是一个普遍存在的挑战，尤其对于需要大量主观评估的任务（如认知状态分类）而言。本研究提出一种半监督学习框架，引入了一种新颖的多视角伪标注方法，该方法利用声学和语言学特征共同筛选出置信度最高的数据用于训练分类模型。在声学层面，通过多个音频编码器生成的嵌入向量计算弗雷歇音频距离，将未标注数据与已标注数据进行对比。在语言学层面，我们引导大语言模型对自动语音识别转录文本进行修订，并基于所提出的任务特定知识预测标签。当两个来源的伪标签一致时，该数据被判定为高置信度数据，而不匹配的数据则视为低置信度数据。随后训练一个双模态分类器，对低置信度数据进行迭代标注，直至满足预设标准。我们在情感识别和痴呆症检测任务上评估了该半监督学习框架。实验结果表明，仅使用30%标注数据时，本方法达到了与全监督学习相当的性能，并显著优于两个选定的基线模型。