The Lip Reading Sentences-3 (LRS3) benchmark has primarily been the focus of intense research in visual speech recognition (VSR) during the last few years. As a result, there is an increased risk of overfitting to its excessively used test set, which is only one hour duration. To alleviate this issue, we build a new VSR test set named WildVSR, by closely following the LRS3 dataset creation processes. We then evaluate and analyse the extent to which the current VSR models generalize to the new test data. We evaluate a broad range of publicly available VSR models and find significant drops in performance on our test set, compared to their corresponding LRS3 results. Our results suggest that the increase in word error rates is caused by the models inability to generalize to slightly harder and in the wild lip sequences than those found in the LRS3 test set. Our new test benchmark is made public in order to enable future research towards more robust VSR models.

翻译：唇读句子-3（LRS3）基准测试在过去几年中一直是视觉语音识别（VSR）领域研究的核心焦点。然而，由于过度依赖仅有一小时时长的测试集，模型存在过拟合风险。为缓解这一问题，我们严格遵循LRS3数据集构建流程，建立了一个名为WildVSR的新VSR测试集。通过评估当前主流VSR模型在新测试数据上的泛化能力，我们发现广泛使用的公开VSR模型在该测试集上的性能较之LRS3结果显著下降。实验结果表明，词错误率的上升源于模型难以泛化至相较LRS3测试集中唇动序列更具挑战性的真实场景案例。为促进未来更鲁棒VSR模型的研究，我们已将新测试基准开源。