Automatic speech recognition (ASR) has gained remarkable successes thanks to recent advances of deep learning, but it usually degrades significantly under real-world noisy conditions. Recent works introduce speech enhancement (SE) as front-end to improve speech quality, which is proved effective but may not be optimal for downstream ASR due to speech distortion problem. Based on that, latest works combine SE and currently popular self-supervised learning (SSL) to alleviate distortion and improve noise robustness. Despite the effectiveness, the speech distortion caused by conventional SE still cannot be cleared out. In this paper, we propose a self-supervised framework named Wav2code to implement a feature-level SE with reduced distortions for noise-robust ASR. First, in pre-training stage the clean speech representations from SSL model are sent to lookup a discrete codebook via nearest-neighbor feature matching, the resulted code sequence are then exploited to reconstruct the original clean representations, in order to store them in codebook as prior. Second, during finetuning we propose a Transformer-based code predictor to accurately predict clean codes by modeling the global dependency of input noisy representations, which enables discovery and restoration of high-quality clean representations with reduced distortions. Furthermore, we propose an interactive feature fusion network to combine original noisy and the restored clean representations to consider both fidelity and quality, resulting in more informative features for downstream ASR. Finally, experiments on both synthetic and real noisy datasets demonstrate that Wav2code can solve the speech distortion and improve ASR performance under various noisy conditions, resulting in stronger robustness.

翻译：自动语音识别（ASR）得益于深度学习的近期进展取得了显著成功，但在实际噪声条件下通常会大幅性能下降。近期研究将语音增强（SE）作为前端以改善语音质量，虽被证明有效，但因语音失真问题可能无法为下游ASR提供最优性能。基于此，最新研究将SE与当前流行的自监督学习（SSL）结合以缓解失真并提升噪声鲁棒性。然而，尽管有效，传统SE导致的语音失真仍无法完全消除。本文提出名为Wav2code的自监督框架，通过实现特征级SE并降低失真以构建噪声鲁棒性ASR。首先，在预训练阶段，通过最近邻特征匹配将SSL模型的干净语音表示映射至离散码本进行查找，所得码序列随后用于重构原始干净表示，从而将其作为先验存储于码本中。其次，微调阶段提出基于Transformer的码预测器，通过建模输入噪声表示的全局依赖关系精确预测干净码，从而发现并恢复高质量、低失真的干净表示。此外，提出交互式特征融合网络以结合原始噪声表示与恢复的干净表示，兼顾保真度与质量，为下游ASR生成更具信息量的特征。最后，在合成和真实噪声数据集上的实验表明，Wav2code能解决语音失真问题，并在多种噪声条件下提升ASR性能，从而实现更强的鲁棒性。