Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech into normal speech by improving the intelligibility and naturalness. This is a challenging task especially for patients with severe dysarthria and speaking in complex, noisy acoustic environments. To address these challenges, we propose a novel multi-modal framework to utilize visual information, e.g., lip movements, in DSR as extra clues for reconstructing the highly abnormal pronunciations. The multi-modal framework consists of: (i) a multi-modal encoder to extract robust phoneme embeddings from dysarthric speech with auxiliary visual features; (ii) a variance adaptor to infer the normal phoneme duration and pitch contour from the extracted phoneme embeddings; (iii) a speaker encoder to encode the speaker's voice characteristics; and (iv) a mel-decoder to generate the reconstructed mel-spectrogram based on the extracted phoneme embeddings, prosodic features and speaker embeddings. Both objective and subjective evaluations conducted on the commonly used UASpeech corpus show that our proposed approach can achieve significant improvements over baseline systems in terms of speech intelligibility and naturalness, especially for the speakers with more severe symptoms. Compared with original dysarthric speech, the reconstructed speech achieves 42.1\% absolute word error rate reduction for patients with more severe dysarthria levels.

翻译：构音障碍语音重建旨在通过提高可懂度和自然度，将构音障碍语音转换为正常语音。这是一项具有挑战性的任务，尤其对于患有严重构音障碍且在复杂嘈杂声学环境中说话的患者而言。为应对这些挑战，我们提出了一种新颖的多模态框架，利用视觉信息（例如唇部运动）作为额外线索来重建高度异常的发音。该多模态框架由以下部分组成：(i) 多模态编码器，用于从构音障碍语音中结合辅助视觉特征提取稳健的音素嵌入；(ii) 方差适配器，用于从提取的音素嵌入中推断正常音素时长和基频轮廓；(iii) 说话人编码器，用于编码说话人的嗓音特征；以及(iv) 梅尔解码器，用于根据提取的音素嵌入、韵律特征和说话人嵌入生成重建的梅尔频谱图。在广泛使用的UASpeech语料库上进行的客观和主观评估均表明，我们提出的方法在语音可懂度和自然度方面相比基线系统取得了显著提升，尤其对于症状更严重的说话人。与原始构音障碍语音相比，对于构音障碍程度更严重的患者，重建语音实现了42.1%的绝对词错误率降低。