Aiming to improve the Automatic Speech Recognition (ASR) outputs with a post-processing step, ASR error correction (EC) techniques have been widely developed due to their efficiency in using parallel text data. Previous works mainly focus on using text or/ and speech data, which hinders the performance gain when not only text and speech information, but other modalities, such as visual information are critical for EC. The challenges are mainly two folds: one is that previous work fails to emphasize visual information, thus rare exploration has been studied. The other is that the community lacks a high-quality benchmark where visual information matters for the EC models. Therefore, this paper provides 1) simple yet effective methods, namely gated fusion and image captions as prompts to incorporate visual information to help EC; 2) large-scale benchmark datasets, namely Visual-ASR-EC, where each item in the training data consists of visual, speech, and text information, and the test data are carefully selected by human annotators to ensure that even humans could make mistakes when visual information is missing. Experimental results show that using captions as prompts could effectively use the visual information and surpass state-of-the-art methods by upto 1.2% in Word Error Rate(WER), which also indicates that visual information is critical in our proposed Visual-ASR-EC dataset

翻译：旨在通过后处理步骤改进自动语音识别（ASR）输出，ASR错误纠正（EC）技术因其在利用并行文本数据方面的高效性而得到广泛应用。以往研究主要集中于使用文本和/或语音数据，这导致当不仅文本和语音信息，而且其他模态（如视觉信息）对EC至关重要时，性能提升受到阻碍。挑战主要来自两方面：一是以往工作未能强调视觉信息，因此相关探索较为稀少；二是学术界缺乏一个高质量基准，在该基准中视觉信息对EC模型具有重要作用。因此，本文提供：1）简单有效的方法，即门控融合和图像描述作为提示，以整合视觉信息辅助EC；2）大规模基准数据集Visual-ASR-EC，其中训练数据的每个条目包含视觉、语音和文本信息，测试数据则由人工标注员精心挑选，以确保即使人类在缺乏视觉信息时也可能犯错。实验结果表明，将图像描述作为提示能够有效利用视觉信息，并在词错误率（WER）上超过最先进方法高达1.2%，这也表明视觉信息在我们提出的Visual-ASR-EC数据集中至关重要。