Recent studies have proposed the use of Text-To-Speech (TTS) systems to automatically synthesise speech test cases on a scale and uncover a large number of failures in ASR systems. However, the failures uncovered by synthetic test cases may not reflect the actual performance of an ASR system when it transcribes human audio, which we refer to as false alarms. Given a failed test case synthesised from TTS systems, which consists of TTS-generated audio and the corresponding ground truth text, we feed the human audio stating the same text to an ASR system. If human audio can be correctly transcribed, an instance of a false alarm is detected. In this study, we investigate false alarm occurrences in five popular ASR systems using synthetic audio generated from four TTS systems and human audio obtained from two commonly used datasets. Our results show that the least number of false alarms is identified when testing Deepspeech, and the number of false alarms is the highest when testing Wav2vec2. On average, false alarm rates range from 21% to 34% in all five ASR systems. Among the TTS systems used, Google TTS produces the least number of false alarms (17%), and Espeak TTS produces the highest number of false alarms (32%) among the four TTS systems. Additionally, we build a false alarm estimator that flags potential false alarms, which achieves promising results: a precision of 98.3%, a recall of 96.4%, an accuracy of 98.5%, and an F1 score of 97.3%. Our study provides insight into the appropriate selection of TTS systems to generate high-quality speech to test ASR systems. Additionally, a false alarm estimator can be a way to minimise the impact of false alarms and help developers choose suitable test inputs when evaluating ASR systems. The source code used in this paper is publicly available on GitHub at https://github.com/julianyonghao/FAinASRtest.

翻译：近期研究提出使用文本转语音（TTS）系统自动大规模合成语音测试用例，以发现自动语音识别（ASR）系统中的大量故障。然而，合成测试用例所揭露的故障可能无法反映ASR系统转录人类语音时的实际性能，这类情况被称为误报。对于TTS系统合成的失败测试用例（包含TTS生成的音频及对应真实文本），我们将朗读相同文本的人类语音输入ASR系统。若人类语音能被正确转录，则检测到一次误报实例。本研究通过四种TTS系统生成的合成音频与来自两个常用数据集的人类音频，在五个主流ASR系统中探究误报发生情况。结果表明：当测试Deepspeech时识别到最少误报，而测试Wav2vec2时误报数量最多。五个ASR系统的平均误报率介于21%至34%之间。在所使用的TTS系统中，Google TTS产生最少误报（17%），而Espeak TTS在四类系统中误报率最高（32%）。此外，我们构建了一个误报估计器用于标记潜在误报，取得了优异效果：精确率达98.3%，召回率达96.4%，准确率达98.5%，F1分数达97.3%。本研究为选择适宜的TTS系统生成高质量语音以测试ASR系统提供了启示。同时，误报估计器可作为降低误报影响的有效手段，帮助开发者在评估ASR系统时选择合适的测试输入。本文使用的源代码已公开于GitHub：https://github.com/julianyonghao/FAinASRtest。