In this paper, we study the transferability of ImageNet spatial and Kinetics spatio-temporal representations to multi-label Movie Trailer Genre Classification (MTGC). In particular, we present an extensive evaluation of the transferability of ConvNet and Transformer models pretrained on ImageNet and Kinetics to Trailers12k, a new manually-curated movie trailer dataset composed of 12,000 videos labeled with 10 different genres and associated metadata. We analyze different aspects that can influence transferability, such as frame rate, input video extension, and spatio-temporal modeling. In order to reduce the spatio-temporal structure gap between ImageNet/Kinetics and Trailers12k, we propose Dual Image and Video Transformer Architecture (DIViTA), which performs shot detection so as to segment the trailer into highly correlated clips, providing a more cohesive input for pretrained backbones and improving transferability (a 1.83% increase for ImageNet and 3.75% for Kinetics). Our results demonstrate that representations learned on either ImageNet or Kinetics are comparatively transferable to Trailers12k. Moreover, both datasets provide complementary information that can be combined to improve classification performance (a 2.91% gain compared to the top single pretraining). Interestingly, using lightweight ConvNets as pretrained backbones resulted in only a 3.46% drop in classification performance compared with the top Transformer while requiring only 11.82% of its parameters and 0.81% of its FLOPS.

翻译：本文研究了ImageNet空间表示与Kinetics时空表示向多标签电影预告片类型分类（MTGC）的迁移能力。具体而言，我们对基于ImageNet和Kinetics预训练的ConvNet与Transformer模型在Trailers12k数据集上的迁移性能进行了全面评估。Trailers12k是一个新的人工标注电影预告片数据集，包含12,000个视频片段，标注了10种类型及其相关元数据。我们分析了可能影响迁移性能的多个因素，包括帧率、输入视频扩展方式以及时空建模方法。为缩小ImageNet/Kinetics与Trailers12k之间的时空结构差异，我们提出了双图像与视频Transformer架构（DIViTA），该架构通过镜头检测将预告片分割为高度相关的片段，为预训练骨干网络提供更连贯的输入，从而提升迁移性能（ImageNet提升1.83%，Kinetics提升3.75%）。实验结果表明，在ImageNet或Kinetics上学习到的表示均可有效迁移至Trailers12k。此外，两个数据集提供互补信息，可结合使用以提升分类性能（相较于单一最优预训练方法提升2.91%）。值得注意的是，采用轻量级ConvNet作为预训练骨干网络时，虽然参数量仅为最优Transformer的11.82%、计算量仅为0.81%，但分类性能仅下降3.46%。