Lack of audio-video synchronization is a common problem during television broadcasts and video conferencing, leading to an unsatisfactory viewing experience. A widely accepted paradigm is to create an error detection mechanism that identifies the cases when audio is leading or lagging. We propose ModEFormer, which independently extracts audio and video embeddings using modality-specific transformers. Different from the other transformer-based approaches, ModEFormer preserves the modality of the input streams which allows us to use a larger batch size with more negative audio samples for contrastive learning. Further, we propose a trade-off between the number of negative samples and number of unique samples in a batch to significantly exceed the performance of previous methods. Experimental results show that ModEFormer achieves state-of-the-art performance, 94.5% for LRS2 and 90.9% for LRS3. Finally, we demonstrate how ModEFormer can be used for offset detection for test clips.

翻译：音视频不同步是电视广播和视频会议中的常见问题，会导致用户观看体验不佳。现有主流方法通常建立错误检测机制，用于识别音频超前或滞后的情况。我们提出ModEFormer方法，通过模态特定Transformer独立提取音频和视频嵌入。与其他基于Transformer的方法不同，ModEFormer保留了输入流的模态特性，从而能在对比学习中使用更大的批次尺寸及更多负音频样本。此外，我们提出在批处理中平衡负样本数量与唯一样本数量，使方法性能显著超越先前方法。实验结果表明，ModEFormer在LRS2和LRS3数据集上分别达到94.5%和90.9%的准确率，实现了当前最优性能。最后，我们展示了ModEFormer在测试片段偏移检测中的应用方法。