Representing speech as discretized units has numerous benefits in supporting downstream spoken language processing tasks. However, the approach has been less explored in speech synthesis of tonal languages like Mandarin Chinese. Our preliminary experiments on Chinese speech synthesis reveal the issue of "tone shift", where a synthesized speech utterance contains correct base syllables but incorrect tones. To address the issue, we propose the ToneUnit framework, which leverages annotated data with tone labels as CTC supervision to learn tone-aware discrete speech units for Mandarin Chinese speech. Our findings indicate that the discrete units acquired through the TonUnit resolve the "tone shift" issue in synthesized Chinese speech and yield favorable results in English synthesis. Moreover, the experimental results suggest that finite scalar quantization enhances the effectiveness of ToneUnit. Notably, ToneUnit can work effectively even with minimal annotated data.

翻译：将语音表示为离散单元在支持下游口语处理任务方面具有诸多优势。然而，该方法在普通话等声调语言的语音合成中尚未得到充分探索。我们在中文语音合成上的初步实验揭示了"声调偏移"问题，即合成语音包含正确的基音节但声调错误。为解决此问题，我们提出了ToneUnit框架，该框架利用带有声调标注的数据作为CTC监督，以学习普通话的声调感知离散语音单元。我们的研究结果表明，通过ToneUnit获取的离散单元解决了合成中文语音中的"声调偏移"问题，并在英语合成中取得了良好效果。此外，实验结果表明有限标量量化增强了ToneUnit的有效性。值得注意的是，即使标注数据量极少，ToneUnit也能有效工作。