The rapid advancement of speech synthesis technologies, including text-to-speech (TTS) and voice conversion (VC), has intensified security and privacy concerns related to voice cloning. Recent defenses attempt to prevent unauthorized cloning by embedding protective perturbations into speech to obscure speaker identity while maintaining intelligibility. However, adversaries can apply advanced purification techniques to remove these perturbations, recover authentic acoustic characteristics, and regenerate cloneable voices. Despite the growing realism of such attacks, the robustness of existing defenses under adaptive purification remains insufficiently studied. Most existing purification methods are designed to counter adversarial noise in automatic speech recognition (ASR) systems rather than speaker verification or voice cloning pipelines. As a result, they fail to suppress the fine-grained acoustic cues that define speaker identity and are often ineffective against speaker verification attacks (SVA). To address these limitations, we propose Diffusion-Bridge (VocalBridge), a purification framework that learns a latent mapping from perturbed to clean speech in the EnCodec latent space. Using a time-conditioned 1D U-Net with a cosine noise schedule, the model enables efficient, transcript-free purification while preserving speaker-discriminative structure. We further introduce a Whisper-guided phoneme variant that incorporates lightweight temporal guidance without requiring ground-truth transcripts. Experimental results show that our approach consistently outperforms existing purification methods in recovering cloneable voices from protected speech. Our findings demonstrate the fragility of current perturbation-based defenses and highlight the need for more robust protection mechanisms against evolving voice-cloning and speaker verification threats.

翻译：语音合成技术（包括文本到语音转换和语音转换）的快速发展加剧了与语音克隆相关的安全和隐私担忧。近期防御方法试图通过在语音中嵌入保护性扰动来防止未经授权的克隆，以掩盖说话人身份同时保持可懂度。然而，攻击者可应用先进的净化技术消除这些扰动，恢复真实声学特征并再生可克隆语音。尽管此类攻击的真实性日益增强，现有防御在自适应净化下的鲁棒性仍未得到充分研究。大多数现有净化方法旨在对抗自动语音识别系统中的对抗性噪声，而非针对说话人验证或语音克隆流程。因此，这些方法无法抑制定义说话人身份的细粒度声学线索，且对说话人验证攻击往往无效。为解决这些局限，我们提出扩散桥（VocalBridge）净化框架，该框架在EnCodec潜在空间中学习从扰动语音到纯净语音的潜在映射。通过采用具有余弦噪声调度的时间条件一维U-Net，该模型能够实现高效、无需转录文本的净化，同时保留说话人判别性结构。我们进一步提出Whisper引导的音素变体，该变体融入轻量级时序引导而无需真实转录文本。实验结果表明，我们的方法在从受保护语音中恢复可克隆语音方面持续优于现有净化方法。本研究揭示了当前基于扰动的防御机制的脆弱性，并强调需要针对不断演变的语音克隆和说话人验证威胁开发更鲁棒的保护机制。