Neural speech codec has recently gained widespread attention in generative speech modeling domains, like voice conversion, text-to-speech synthesis, etc. However, ensuring high-fidelity audio reconstruction of speech codecs under low bitrate remains an open and challenging issue. In this paper, we propose PromptCodec, a novel end-to-end neural speech codec using feature-aware prompt encoders based on disentangled representation learning. By incorporating prompt encoders to capture representations of additional input prompts, PromptCodec can distribute the speech information requiring processing and enhance its representation capabilities. Moreover, a simple yet effective adaptive feature weighted fusion approach is introduced to integrate features of different encoders. Meanwhile, we propose a novel disentangled representation learning strategy based on structure similarity index measure to optimize PromptCodec's encoders to ensure their efficiency, thereby further improving the performance of PromptCodec. Experiments on LibriTTS demonstrate that our proposed PromptCodec consistently outperforms state-of-the-art neural speech codec models under all different bitrate conditions while achieving superior performance with low bitrates.

翻译：神经语音编解码器近来在语音转换、文本到语音合成等生成式语音建模领域受到广泛关注。然而，在低比特率下确保语音编解码器的高保真音频重建仍然是一个开放且充满挑战的问题。本文提出PromptCodec——一种新颖的端到端神经语音编解码器，它基于解耦表示学习并采用特征感知提示编码器。通过引入提示编码器来捕获额外输入提示的表示，PromptCodec能够分配待处理的语音信息并增强其表示能力。此外，我们引入了一种简单而有效的自适应特征加权融合方法，用于整合不同编码器的特征。同时，我们提出了一种基于结构相似性指标测量的新型解耦表示学习策略，以优化PromptCodec的编码器，确保其效率，从而进一步提升PromptCodec的性能。在LibriTTS上的实验表明，我们提出的PromptCodec在所有不同比特率条件下均持续优于最先进的神经语音编解码器模型，并在低比特率下实现了卓越性能。