Zero-shot speaker cloning aims to synthesize speech for any target speaker unseen during TTS system building, given only a single speech reference of the speaker at hand. Although more practical in real applications, the current zero-shot methods still produce speech with undesirable naturalness and speaker similarity. Moreover, endowing the target speaker with arbitrary speaking styles in the zero-shot setup has not been considered. This is because the unique challenge of zero-shot speaker and style cloning is to learn the disentangled speaker and style representations from only short references representing an arbitrary speaker and an arbitrary style. To address this challenge, we propose U-Style, which employs Grad-TTS as the backbone, particularly cascading a speaker-specific encoder and a style-specific encoder between the text encoder and the diffusion decoder. Thus, leveraging signal perturbation, U-Style is explicitly decomposed into speaker- and style-specific modeling parts, achieving better speaker and style disentanglement. To improve unseen speaker and style modeling ability, these two encoders conduct multi-level speaker and style modeling by skip-connected U-nets, incorporating the representation extraction and information reconstruction process. Besides, to improve the naturalness of synthetic speech, we adopt mean-based instance normalization and style adaptive layer normalization in these encoders to perform representation extraction and condition adaptation, respectively. Experiments show that U-Style significantly surpasses the state-of-the-art methods in unseen speaker cloning regarding naturalness and speaker similarity. Notably, U-Style can transfer the style from an unseen source speaker to another unseen target speaker, achieving flexible combinations of desired speaker timbre and style in zero-shot voice cloning.
翻译:零样本说话人克隆旨在仅凭目标说话人的单一语音参考片段,为其合成语音(该说话人在TTS系统构建阶段未曾出现)。尽管在实际应用中更具实用性,现有零样本方法生成的语音在自然度和说话人相似性方面仍不尽人意。此外,在零样本场景下赋予目标说话人任意说话风格的问题尚未被考虑。这是因为零样本说话人与风格克隆的核心挑战在于:仅通过代表任意说话人和任意风格的短参考片段,学习分离的说话人与风格表征。为应对这一挑战,我们提出U-Style,以Grad-TTS为骨干架构,在文本编码器与扩散解码器之间级联了说话人专用编码器和风格专用编码器。通过引入信号扰动机制,U-Style被显式分解为说话人特定与风格特定建模模块,实现了更优的说话人与风格解耦。为提升未见说话人与风格的建模能力,这两个编码器采用跳连U型网络进行多层级说话人与风格建模,融合了表征提取与信息重建过程。此外,为改善合成语音自然度,我们在这些编码器中分别采用基于均值的实例归一化与风格自适应层归一化,分别执行表征提取与条件适配。实验表明,U-Style在未见说话人克隆的自然度和说话人相似性方面显著超越了现有最优方法。值得注意的是,U-Style能将风格从未见源说话人迁移至另一个未见目标说话人,在零样本语音克隆中实现了目标说话人音色与所需风格的灵活组合。