Video anomaly detection (VAD) is a significant computer vision problem. Existing deep neural network (DNN) based VAD methods mostly follow the route of frame reconstruction or frame prediction. However, the lack of mining and learning of higher-level visual features and temporal context relationships in videos limits the further performance of these two approaches. Inspired by video codec theory, we introduce a brand-new VAD paradigm to break through these limitations: First, we propose a new task of video event restoration based on keyframes. Encouraging DNN to infer missing multiple frames based on video keyframes so as to restore a video event, which can more effectively motivate DNN to mine and learn potential higher-level visual features and comprehensive temporal context relationships in the video. To this end, we propose a novel U-shaped Swin Transformer Network with Dual Skip Connections (USTN-DSC) for video event restoration, where a cross-attention and a temporal upsampling residual skip connection are introduced to further assist in restoring complex static and dynamic motion object features in the video. In addition, we propose a simple and effective adjacent frame difference loss to constrain the motion consistency of the video sequence. Extensive experiments on benchmarks demonstrate that USTN-DSC outperforms most existing methods, validating the effectiveness of our method.

翻译：视频异常检测（VAD）是计算机视觉领域的重要问题。现有基于深度神经网络（DNN）的VAD方法大多遵循帧重建或帧预测的路线。然而，由于缺乏对视频中更高层视觉特征和时序上下文关系的挖掘与学习，这两种方法的性能提升受到限制。受视频编解码理论的启发，我们提出了一种全新的VAD范式以突破这些限制：首先，我们提出了基于关键帧的视频事件恢复新任务。通过鼓励DNN根据视频关键帧推断缺失的多个帧以恢复视频事件，这能够更有效地激励DNN挖掘和学习视频中潜在的高层视觉特征及全面的时序上下文关系。为此，我们提出了一种具有双跳跃连接的新型U型Swin Transformer网络（USTN-DSC）用于视频事件恢复，其中引入了交叉注意力机制和时序上采样残差跳跃连接，以进一步辅助恢复视频中复杂的静态与动态运动目标特征。此外，我们提出了一种简单有效的相邻帧差异损失函数，用于约束视频序列的运动一致性。在基准数据集上的大量实验表明，USTN-DSC的性能优于现有大多数方法，验证了我们方法的有效性。