Recent years have witnessed wider adoption of Automated Speech Recognition (ASR) techniques in various domains. Consequently, evaluating and enhancing the quality of ASR systems is of great importance. This paper proposes ASDF, an Automated Speech Recognition Differential Testing Framework for testing ASR systems. ASDF extends an existing ASR testing tool, the CrossASR++, which synthesizes test cases from a text corpus. However, CrossASR++ fails to make use of the text corpus efficiently and provides limited information on how the failed test cases can improve ASR systems. To address these limitations, our tool incorporates two novel features: (1) a text transformation module to boost the number of generated test cases and uncover more errors in ASR systems and (2) a phonetic analysis module to identify on which phonemes the ASR system tend to produce errors. ASDF generates more high-quality test cases by applying various text transformation methods (e.g., change tense) to the texts in failed test cases. By doing so, ASDF can utilize a small text corpus to generate a large number of audio test cases, something which CrossASR++ is not capable of. In addition, ASDF implements more metrics to evaluate the performance of ASR systems from multiple perspectives. ASDF performs phonetic analysis on the identified failed test cases to identify the phonemes that ASR systems tend to transcribe incorrectly, providing useful information for developers to improve ASR systems. The demonstration video of our tool is made online at https://www.youtube.com/watch?v=DzVwfc3h9As. The implementation is available at https://github.com/danielyuenhx/asdf-differential-testing.

翻译：近年来，自动语音识别（ASR）技术在多个领域得到广泛应用。因此，评估和提升ASR系统的质量至关重要。本文提出ASDF，一种用于测试ASR系统的自动语音识别差分测试框架。ASDF扩展了现有ASR测试工具CrossASR++，该工具通过文本语料库合成测试用例。然而，CrossASR++未能有效利用文本语料库，且对于失败测试用例如何改进ASR系统提供的信息有限。为克服这些局限，我们的工具引入两个新特性：（1）文本转换模块，用于增加生成的测试用例数量并揭示ASR系统中更多错误；（2）语音分析模块，用于识别ASR系统易出错的音素。ASDF通过对失败测试用例中的文本应用多种文本转换方法（如改变时态）来生成更多高质量测试用例。由此，ASDF可以利用小型文本语料库生成大量音频测试用例，而CrossASR++不具备这一能力。此外，ASDF实现了更多指标以从多角度评估ASR系统性能。ASDF对识别出的失败测试用例进行语音分析，确定ASR系统易转录错误的音素，为开发者改进ASR系统提供有用信息。本工具演示视频在线发布地址为https://www.youtube.com/watch?v=DzVwfc3h9As，实现代码见https://github.com/danielyuenhx/asdf-differential-testing。