Currently, many multi-speaker speech synthesis and voice conversion systems address speaker variations with an embedding vector. Modeling it directly allows new voices outside of training data to be synthesized. GMM based approaches such as Tacospawn are favored in literature for this generation task, but there are still some limitations when difficult conditionings are involved. In this paper, we propose VoiceLens, a semi-supervised flow-based approach, to model speaker embedding distributions for multi-conditional speaker generation. VoiceLens maps speaker embeddings into a combination of independent attributes and residual information. It allows new voices associated with certain attributes to be \textit{generated} for existing TTS models, and attributes of known voices to be meaningfully \textit{edited}. We show in this paper, VoiceLens displays an unconditional generation capacity that is similar to Tacospawn while obtaining higher controllability and flexibility when used in a conditional manner. In addition, we show synthesizing less noisy speech from known noisy speakers without re-training the TTS model is possible via solely editing their embeddings with a SNR conditioned VoiceLens model. Demos are available at sos1sos2sixteen.github.io/voicelens.

翻译：当前，许多多说话人语音合成与语音转换系统通过嵌入向量处理说话人差异。直接对其建模能合成训练数据之外的语音。在文献中，基于高斯混合模型的方法（如Tacospawn）在该生成任务上备受青睐，但在涉及复杂条件约束时仍存在局限性。本文提出VoiceLens——一种半监督流式方法，用于对多条件说话人生成任务中的说话人嵌入分布进行建模。VoiceLens将说话人嵌入映射为独立属性与残差信息的组合，使得现有TTS模型能够生成与特定属性关联的新语音，同时实现对已知语音属性的有意义编辑。实验表明，VoiceLens在无条件生成能力上与Tacospawn相当，但在条件化使用时展现出更高的可控性与灵活性。此外，我们证明仅通过信噪比条件化的VoiceLens模型编辑已知嘈杂说话人的嵌入，即可在不重新训练TTS模型的情况下合成更少噪声的语音。Demo地址：sos1sos2sixteen.github.io/voicelens。