Some recent models for Text-to-Speech synthesis aim to transfer the prosody of a reference utterance to the generated target synthetic speech. This is done by using a learned embedding of the reference utterance, which is used to condition speech generation. During training, the reference utterance is identical to the target utterance. Yet, during synthesis, these models are often used to transfer prosody from a reference that differs from the text or speaker being synthesized. To address this inconsistency, we propose to use a different, but prosodically-related, utterance during training too. We believe this should encourage the model to learn to transfer only those characteristics that the reference and target have in common. If prosody transfer methods do indeed transfer prosody they should be able to be trained in the way we propose. However, results show that a model trained under these conditions performs significantly worse than one trained using the target utterance as a reference. To explain this, we hypothesize that prosody transfer models do not learn a transferable representation of prosody, but rather an utterance-level representation which is highly dependent on both the reference speaker and reference text.

翻译：近期一些文本到语音合成模型旨在将参考语音的韵律迁移至生成的合成语音。该方法通过提取参考语音的嵌入表示来条件化语音生成过程。训练时参考语音与目标语音完全一致，但在合成阶段，这些模型常被用来从与合成文本或说话人不匹配的参考语音中迁移韵律。为解决此不一致性，我们提出在训练阶段也使用韵律相关但不同的语音。这应能促使模型学习仅迁移参考语音与目标语音共有的特征。若韵律迁移方法确实能迁移韵律，则其应能按我们提出的方式进行训练。然而实验结果显示，在此条件下训练的模型性能显著低于使用目标语音作为参考的模型。对此我们提出假设：韵律迁移模型并未习得可迁移的韵律表征，而是获得了高度依赖参考说话人和参考文本的语句级表征。