Zero-shot voice conversion (VC) aims to transform the timbre of a source speaker into any previously unseen target speaker, while preserving the original linguistic content. Despite notable progress, attaining a degree of speaker similarity and naturalness on par with ground truth recordings continues to pose great challenge. In this paper, we propose CTEFM-VC, a zero-shot VC framework that leverages Content-aware Timbre Ensemble modeling and Flow Matching. Specifically, CTEFM-VC disentangles utterances into linguistic content and timbre representations, subsequently utilizing a conditional flow matching model and a vocoder to reconstruct the mel-spectrogram and waveform. To enhance its timbre modeling capability and the naturalness of generated speech, we propose a context-aware timbre ensemble modeling approach that adaptively integrates diverse speaker verification embeddings and enables the joint utilization of linguistic and timbre features through a cross-attention module. Experiments show that our CTEFM-VC system surpasses state-of-the-art VC methods in both speaker similarity and naturalness by at least 18.5% and 7.0%.

翻译：零样本语音转换旨在将源说话人的音色转换为任何先前未见过的目标说话人音色，同时保持原始的语言内容。尽管取得了显著进展，但达到与真实录音相媲美的说话人相似度和自然度仍然是一个巨大的挑战。本文提出CTEFM-VC，一个利用内容感知音色集成建模与流匹配的零样本语音转换框架。具体而言，CTEFM-VC将语音解耦为语言内容和音色表示，随后利用条件流匹配模型和声码器重建梅尔频谱图和波形。为了增强其音色建模能力和生成语音的自然度，我们提出了一种上下文感知的音色集成建模方法，该方法自适应地整合多种说话人验证嵌入，并通过交叉注意力模块实现语言特征与音色特征的联合利用。实验表明，我们的CTEFM-VC系统在说话人相似度和自然度上均超越了最先进的语音转换方法，提升幅度至少分别为18.5%和7.0%。