In recent years, the evolution of end-to-end (E2E) automatic speech recognition (ASR) models has been remarkable, largely due to advances in deep learning architectures like transformer. On top of E2E systems, researchers have achieved substantial accuracy improvement by rescoring E2E model's N-best hypotheses with a phoneme-based model. This raises an interesting question about where the improvements come from other than the system combination effect. We examine the underlying mechanisms driving these gains and propose an efficient joint training approach, where E2E models are trained jointly with diverse modeling units. This methodology does not only align the strengths of both phoneme and grapheme-based models but also reveals that using these diverse modeling units in a synergistic way can significantly enhance model accuracy. Our findings offer new insights into the optimal integration of heterogeneous modeling units in the development of more robust and accurate ASR systems.

翻译：近年来，端到端（E2E）自动语音识别（ASR）模型的发展引人注目，这主要得益于Transformer等深度学习架构的进步。在E2E系统基础上，研究者通过使用基于音素的模型对E2E模型的N-best假设进行重打分，实现了显著的准确率提升。这引发了一个有趣的问题：除了系统组合效应之外，这些改进究竟源于何处？我们研究了驱动这些增益的内在机制，并提出了一种高效的联合训练方法，即使用多样化建模单元对E2E模型进行联合训练。该方法不仅整合了基于音素与基于字素模型的优势，还揭示了以协同方式使用这些多样化建模单元能够显著提升模型准确率。我们的研究结果为在开发更鲁棒、更准确的ASR系统中实现异构建模单元的最优集成提供了新的见解。