In recent years, large-scale pre-trained speech language models (SLMs) have demonstrated remarkable advancements in various generative speech modeling applications, such as text-to-speech synthesis, voice conversion, and speech enhancement. These applications typically involve mapping text or speech inputs to pre-trained SLM representations, from which target speech is decoded. This paper introduces a new approach, SLMGAN, to leverage SLM representations for discriminative tasks within the generative adversarial network (GAN) framework, specifically for voice conversion. Building upon StarGANv2-VC, we add our novel SLM-based WavLM discriminators on top of the mel-based discriminators along with our newly designed SLM feature matching loss function, resulting in an unsupervised zero-shot voice conversion system that does not require text labels during training. Subjective evaluation results show that SLMGAN outperforms existing state-of-the-art zero-shot voice conversion models in terms of naturalness and achieves comparable similarity, highlighting the potential of SLM-based discriminators for related applications.

翻译：近年来，大规模预训练语音语言模型在文本转语音合成、语音转换及语音增强等多种生成式语音建模应用中取得了显著进展。这些应用通常涉及将文本或语音输入映射到预训练的SLM表征，进而从中解码出目标语音。本文提出了一种新方法SLMGAN，旨在生成对抗网络框架中利用SLM表征执行判别任务，具体应用于语音转换。基于StarGANv2-VC，我们在梅尔频谱判别器的基础上新增基于SLM的WavLM判别器，并设计了新的SLM特征匹配损失函数，从而构建了一个无需训练文本标签的无监督零样本语音转换系统。主观评估结果表明，SLMGAN在自然度上优于现有最先进的零样本语音转换模型，并达到了可比的相似度，凸显了基于SLM的判别器在相关应用中的潜力。