We present StreamVC, a streaming voice conversion solution that preserves the content and prosody of any source speech while matching the voice timbre from any target speech. Unlike previous approaches, StreamVC produces the resulting waveform at low latency from the input signal even on a mobile platform, making it applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios. Our design leverages the architecture and training strategy of the SoundStream neural audio codec for lightweight high-quality speech synthesis. We demonstrate the feasibility of learning soft speech units causally, as well as the effectiveness of supplying whitened fundamental frequency information to improve pitch stability without leaking the source timbre information.

翻译：我们提出了StreamVC，一种流式语音转换解决方案，该方案在匹配任意目标语音音色的同时，保留任何源语音的内容和韵律。与以往方法不同，StreamVC即使在移动平台上也能以低延迟从输入信号生成结果波形，因此适用于电话会议和视频通话等实时通信场景，并能解决这些场景下的语音匿名化等应用需求。我们的设计借鉴了SoundStream神经音频编解码器的架构和训练策略，以实现轻量级高质量语音合成。我们证明了因果学习软语音单元的可行性，以及提供白化基频信息在不泄露源语音音色信息的前提下提高音调稳定性的有效性。