Affect is an emotional characteristic encompassing valence, arousal, and intensity, and is a crucial attribute for enabling authentic conversations. While existing text-to-speech (TTS) and speech-to-speech systems rely on strength embedding vectors and global style tokens to capture emotions, these models represent emotions as a component of style or represent them in discrete categories. We propose AffectEcho, an emotion translation model, that uses a Vector Quantized codebook to model emotions within a quantized space featuring five levels of affect intensity to capture complex nuances and subtle differences in the same emotion. The quantized emotional embeddings are implicitly derived from spoken speech samples, eliminating the need for one-hot vectors or explicit strength embeddings. Experimental results demonstrate the effectiveness of our approach in controlling the emotions of generated speech while preserving identity, style, and emotional cadence unique to each speaker. We showcase the language-independent emotion modeling capability of the quantized emotional embeddings learned from a bilingual (English and Chinese) speech corpus with an emotion transfer task from a reference speech to a target speech. We achieve state-of-art results on both qualitative and quantitative metrics.

翻译：情绪是一种涵盖效价、唤醒度和强度的情感特征，是实现真实对话的关键属性。现有文本转语音（TTS）和语音转语音系统依赖强度嵌入向量和全局风格标记来捕获情绪，但这些模型将情绪表示为风格的一部分或离散类别。我们提出AffectEcho情绪翻译模型，该模型采用向量量化码本在量化空间中建模情绪，特征包含五种情绪强度层次，以捕捉相同情绪的复杂细微差异。量化情感嵌入隐式地从语音样本中推导得出，无需使用独热向量或显式强度嵌入。实验结果表明，该方法在控制生成语音情绪的同时，能保持每位说话者独特的身份、风格和情感韵律。我们通过从参考语音到目标语音的情绪迁移任务，展示了基于双语（英语和中文）语音语料库学习的量化情感嵌入的语言无关建模能力。在定性和定量指标上均取得最先进成果。