Learning high-quality video representation has shown significant applications in computer vision and remains challenging. Previous work based on mask autoencoders such as ImageMAE and VideoMAE has proven the effectiveness of learning representations in images and videos through reconstruction strategy in the visual modality. However, these models exhibit inherent limitations, particularly in scenarios where extracting features solely from the visual modality proves challenging, such as when dealing with low-resolution and blurry original videos. Based on this, we propose AV-MaskEnhancer for learning high-quality video representation by combining visual and audio information. Our approach addresses the challenge by demonstrating the complementary nature of audio and video features in cross-modality content. Moreover, our result of the video classification task on the UCF101 dataset outperforms the existing work and reaches the state-of-the-art, with a top-1 accuracy of 98.8% and a top-5 accuracy of 99.9%.

翻译：学习高质量视频表征在计算机视觉领域具有重要应用，但仍面临挑战。先前基于掩码自编码器的研究（如ImageMAE和VideoMAE）已证明通过视觉模态的重建策略学习图像和视频表征的有效性。然而，这些模型存在固有局限性，特别是在仅从视觉模态提取特征较为困难的场景中（例如处理低分辨率和模糊的原始视频时）。基于此，我们提出AV-MaskEnhancer，通过融合视觉和音频信息来学习高质量视频表征。该方法通过证明跨模态内容中音频与视频特征的互补性来解决该挑战。此外，我们在UCF101数据集上的视频分类任务结果超越现有工作并达到最优水平，top-1准确率为98.8%，top-5准确率为99.9%。