We have seen a great progress in video action recognition in recent years. There are several models based on convolutional neural network (CNN) and some recent transformer based approaches which provide top performance on existing benchmarks. In this work, we perform a large-scale robustness analysis of these existing models for video action recognition. We focus on robustness against real-world distribution shift perturbations instead of adversarial perturbations. We propose four different benchmark datasets, HMDB51-P, UCF101-P, Kinetics400-P, and SSv2-P to perform this analysis. We study robustness of six state-of-the-art action recognition models against 90 different perturbations. The study reveals some interesting findings, 1) transformer based models are consistently more robust compared to CNN based models, 2) Pretraining improves robustness for Transformer based models more than CNN based models, and 3) All of the studied models are robust to temporal perturbations for all datasets but SSv2; suggesting the importance of temporal information for action recognition varies based on the dataset and activities. Next, we study the role of augmentations in model robustness and present a real-world dataset, UCF101-DS, which contains realistic distribution shifts, to further validate some of these findings. We believe this study will serve as a benchmark for future research in robust video action recognition.

翻译：近年来，视频动作识别领域取得了显著进展。基于卷积神经网络（CNN）的多种模型以及近期部分基于Transformer的方法，在现有基准测试中展现出顶尖性能。本研究对现有视频动作识别模型进行了大规模鲁棒性分析，重点关注模型对现实分布偏移扰动（而非对抗性扰动）的鲁棒性。为此，我们提出了四个不同的基准数据集：HMDB51-P、UCF101-P、Kinetics400-P 和 SSv2-P，对六种最先进的动作识别模型在90种不同扰动下的鲁棒性展开研究。实验揭示了若干有趣发现：1）基于Transformer的模型在鲁棒性上始终优于基于CNN的模型；2）预训练对Transformer模型鲁棒性的提升幅度大于CNN模型；3）除SSv2数据集外，所有被研究模型对时间扰动均表现出鲁棒性，这表明时间信息在动作识别中的重要性因数据集及活动类型而异。此外，我们还研究了数据增强对模型鲁棒性的作用，并提供了一个包含现实分布偏移的真实数据集UCF101-DS，以进一步验证部分结论。我们相信，本研究将为未来鲁棒视频动作识别的研究提供基准参考。