In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner. In particular, the audio and visual encoders learn to extract features directly from raw pixels and audio waveforms, respectively, which are then fed to conformers and then fusion takes place via a Multi-Layer Perceptron (MLP). The model learns to recognise characters using a combination of CTC and an attention mechanism. We show that end-to-end training, instead of using pre-computed visual features which is common in the literature, the use of a conformer, instead of a recurrent network, and the use of a transformer-based language model, significantly improve the performance of our model. We present results on the largest publicly available datasets for sentence-level speech recognition, Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3), respectively. The results show that our proposed models raise the state-of-the-art performance by a large margin in audio-only, visual-only, and audio-visual experiments.

翻译：在这项工作中,我们提出了一个基于ResNet-18和革命放大变压器(变压器)的混合CTC/注意力模型,可以以端到端方式进行培训,特别是视听编码器学会直接从生像素和声波形状中提取特征,然后将其配给校对者,然后通过多语言感应器(MLP)进行聚合。该模型学会使用CTC和注意机制组合来识别字符。我们显示,最终到终端培训,而不是使用文献中常见的预合成视觉特征,使用校对器,而不是经常网络,以及使用变压器语言模型,大大改进了我们模型的性能。我们分别介绍了用于判词识别的公开最大数据集,即LRS2(LRS2)和LRS3(LRS3)的结果。结果显示,我们提议的模型在视听、视觉、视听和视听试验中以较大幅度提升了艺术状态的性能。