Learning high-quality video representation has shown significant applications in computer vision and remains challenging. Previous work based on mask autoencoders such as ImageMAE and VideoMAE has proven the effectiveness of learning representations in images and videos through reconstruction strategy in the visual modality. However, these models exhibit inherent limitations, particularly in scenarios where extracting features solely from the visual modality proves challenging, such as when dealing with low-resolution and blurry original videos. Based on this, we propose AV-MaskEnhancer for learning high-quality video representation by combining visual and audio information. Our approach addresses the challenge by demonstrating the complementary nature of audio and video features in cross-modality content. Moreover, our result of the video classification task on the UCF101 dataset outperforms the existing work and reaches the state-of-the-art, with a top-1 accuracy of 98.8% and a top-5 accuracy of 99.9%.

翻译：学习高质量的视频表示在计算机视觉中具有重要应用，但仍具挑战性。基于掩码自编码器的先前工作（如ImageMAE和VideoMAE）已证明通过视觉模态中的重构策略学习图像和视频表示的有效性。然而，这些模型存在固有局限性，尤其在仅从视觉模态提取特征具有挑战性的场景中（如处理低分辨率、模糊的原始视频时）表现不足。基于此，我们提出AV-MaskEnhancer，通过融合视觉与音频信息学习高质量视频表示。该方法通过证明跨模态内容中音频与视频特征的互补性来解决上述挑战。此外，我们在UCF101数据集上的视频分类任务结果超越现有工作，达到最先进水平，top-1准确率为98.8%，top-5准确率为99.9%。