Diffusion-based generative models have exhibited powerful generative performance in recent years. However, as many attributes exist in the data distribution and owing to several limitations of sharing the model parameters across all levels of the generation process, it remains challenging to control specific styles for each attribute. To address the above problem, this paper presents decoupled denoising diffusion models (DDDMs) with disentangled representations, which can control the style for each attribute in generative models. We apply DDDMs to voice conversion (VC) tasks to address the challenges of disentangling and controlling each speech attribute (e.g., linguistic information, intonation, and timbre). First, we use a self-supervised representation to disentangle the speech representation. Subsequently, the DDDMs are applied to resynthesize the speech from the disentangled representations for denoising with respect to each attribute. Moreover, we also propose the prior mixup for robust voice style transfer, which uses the converted representation of the mixed style as a prior distribution for the diffusion models. The experimental results reveal that our method outperforms publicly available VC models. Furthermore, we show that our method provides robust generative performance regardless of the model size. Audio samples are available https://hayeong0.github.io/DDDM-VC-demo/.

翻译：近年来，基于扩散的生成模型展现出强大的生成性能。然而，由于数据分布中存在多种属性，且生成过程各层级共享模型参数存在局限性，因此对每个属性进行特定风格控制仍具挑战性。为解决上述问题，本文提出带有解缠表示的解耦去噪扩散模型（DDDMs），该模型能够在生成模型中控制每个属性的风格。我们将DDDMs应用于语音转换（VC）任务，以解决解缠和控制每个语音属性（如语言信息、语调、音色）的挑战。首先，采用自监督表示方法对语音表示进行解缠；随后，应用DDDMs根据解缠表示对每个属性进行去噪并重合成语音。此外，我们还提出用于鲁棒语音风格迁移的先验混合方法，将混合风格的转换表示作为扩散模型的先验分布。实验结果表明，本方法优于公开可用的语音转换模型。进一步证明，无论模型大小如何，该方法均能提供稳健的生成性能。音频样本可访问 https://hayeong0.github.io/DDDM-VC-demo/。