Speech tokenization enables discrete representation and facilitates speech language modeling. However, existing neural codecs capture low-level acoustic features, overlooking the semantic and contextual cues inherent to human speech. While recent efforts introduced semantic representations from self-supervised speech models or incorporated contextual representations from pre-trained language models, challenges remain in aligning and unifying the semantic and contextual representations. We introduce FuseCodec, which unifies acoustic, semantic, and contextual representations through strong cross-modal alignment and globally informed supervision. We propose three complementary techniques: (i) Latent Representation Fusion, integrating semantic and contextual features directly into the encoder latent space for robust and unified representation learning; (ii) Global Semantic-Contextual Supervision, supervising discrete tokens with globally pooled and broadcasted representations to enhance temporal consistency and cross-modal alignment; and (iii) Temporally Aligned Contextual Supervision, strengthening alignment by dynamically matching contextual and speech tokens within a local window for fine-grained token-level supervision. We further introduce FuseCodec-TTS, demonstrating our methodology's applicability to zero-shot speech synthesis. Empirically, FuseCodec achieves state-of-the-art performance in LibriSpeech, surpassing EnCodec, SpeechTokenizer, and DAC in transcription accuracy, perceptual quality, intelligibility, and speaker similarity. Results highlight the effectiveness of contextually and semantically guided tokenization for speech tokenization and downstream tasks. Code and pretrained models are available at https://github.com/mubtasimahasan/FuseCodec.

翻译：语音分词技术能够实现离散表示并促进语音语言建模。然而，现有的神经编解码器主要捕获低层次声学特征，忽略了人类语音固有的语义与上下文线索。尽管近期研究引入了来自自监督语音模型的语义表示，或融合了预训练语言模型的上下文表示，但在对齐与统一语义和上下文表示方面仍面临挑战。我们提出了FuseCodec，该方法通过强跨模态对齐和全局信息监督，统一了声学、语义和上下文表示。我们提出了三种互补的技术：(i) 潜在表示融合，将语义和上下文特征直接集成到编码器潜在空间中，以实现鲁棒且统一的表示学习；(ii) 全局语义-上下文监督，利用全局池化与广播的表示对离散令牌进行监督，以增强时序一致性与跨模态对齐；(iii) 时序对齐的上下文监督，通过在局部窗口内动态匹配上下文与语音令牌，以加强细粒度令牌级监督的对齐。我们进一步提出了FuseCodec-TTS，展示了该方法在零样本语音合成中的适用性。实验表明，FuseCodec在LibriSpeech数据集上取得了最先进的性能，在转录准确率、感知质量、可懂度和说话人相似度方面均超越了EnCodec、SpeechTokenizer和DAC。结果凸显了基于上下文与语义引导的分词方法在语音分词及下游任务中的有效性。代码与预训练模型发布于 https://github.com/mubtasimahasan/FuseCodec。