Automatic speech recognition (ASR) has gained a remarkable success thanks to recent advances of deep learning, but it usually degrades significantly under real-world noisy conditions. Recent works introduce speech enhancement (SE) as front-end to improve speech quality, which is proved effective but may not be optimal for downstream ASR due to speech distortion problem. Based on that, latest works combine SE and currently popular self-supervised learning (SSL) to alleviate distortion and improve noise robustness. Despite the effectiveness, the speech distortion caused by conventional SE still cannot be completely eliminated. In this paper, we propose a self-supervised framework named Wav2code to implement a generalized SE without distortions for noise-robust ASR. First, in pre-training stage the clean speech representations from SSL model are sent to lookup a discrete codebook via nearest-neighbor feature matching, the resulted code sequence are then exploited to reconstruct the original clean representations, in order to store them in codebook as prior. Second, during finetuning we propose a Transformer-based code predictor to accurately predict clean codes by modeling the global dependency of input noisy representations, which enables discovery and restoration of high-quality clean representations without distortions. Furthermore, we propose an interactive feature fusion network to combine original noisy and the restored clean representations to consider both fidelity and quality, resulting in even more informative features for downstream ASR. Finally, experiments on both synthetic and real noisy datasets demonstrate that Wav2code can solve the speech distortion and improve ASR performance under various noisy conditions, resulting in stronger robustness.

翻译：自动语音识别（ASR）因深度学习的进展取得了显著成功，但在真实世界噪声条件下性能通常会大幅下降。近期研究引入语音增强（SE）作为前端模块来提升语音质量，虽然被证明有效，但由于语音失真问题可能对下游ASR并非最优。在此基础上，最新工作将SE与当前流行的自监督学习（SSL）相结合，以缓解失真并提升噪声鲁棒性。尽管有效，但传统SE导致的语音失真仍无法完全消除。本文提出一种名为Wav2code的自监督框架，旨在为噪声鲁棒ASR实现无失真的广义语音增强。首先，在预训练阶段，SSL模型提取的干净语音表示通过最近邻特征匹配查找离散码本，生成的码序列随后用于重构原始干净表示，从而将其作为先验存储在码本中。其次，在微调阶段，我们提出基于Transformer的码预测器，通过建模输入噪声表示的全局依赖关系精准预测干净码，从而发现并恢复高质量的无失真干净表示。此外，我们提出交互式特征融合网络，结合原始噪声表示与恢复的干净表示，兼顾保真度与质量，为下游ASR生成信息更丰富的特征。最后，在合成与真实噪声数据集上的实验表明，Wav2code能解决语音失真问题，并在多种噪声条件下提升ASR性能，实现更强的鲁棒性。