Contrastive language-image pretraining (CLIP) has demonstrated remarkable success in various image tasks. However, how to extend CLIP with effective temporal modeling is still an open and crucial problem. Existing factorized or joint spatial-temporal modeling trades off between the efficiency and performance. While modeling temporal information within straight through tube is widely adopted in literature, we find that simple frame alignment already provides enough essence without temporal attention. To this end, in this paper, we proposed a novel Implicit Learnable Alignment (ILA) method, which minimizes the temporal modeling effort while achieving incredibly high performance. Specifically, for a frame pair, an interactive point is predicted in each frame, serving as a mutual information rich region. By enhancing the features around the interactive point, two frames are implicitly aligned. The aligned features are then pooled into a single token, which is leveraged in the subsequent spatial self-attention. Our method allows eliminating the costly or insufficient temporal self-attention in video. Extensive experiments on benchmarks demonstrate the superiority and generality of our module. Particularly, the proposed ILA achieves a top-1 accuracy of 88.7% on Kinetics-400 with much fewer FLOPs compared with Swin-L and ViViT-H. Code is released at https://github.com/Francis-Rings/ILA .

翻译：对比语言-图像预训练（CLIP）已在多种图像任务中展现出卓越性能。然而，如何为CLIP扩展有效的时序建模仍是一个关键且待解决的核心问题。现有的因子化或联合时空建模方法需要在效率与性能之间权衡取舍。尽管沿时间维度直通管道的时序建模方法在文献中被广泛采用，但本研究发现，无需时序注意力机制，简单的帧对齐已能提供足够的语义本质。基于此，本文提出一种新颖的隐式可学习对齐（ILA）方法，该方法在最小化时序建模开销的同时实现了极高性能。具体而言，对于帧对，我们分别在每帧中预测一个交互点作为互信息富集区域，通过增强交互点周围特征实现两帧的隐式对齐。对齐后的特征被池化为单一令牌，用于后续的空间自注意力计算。本方法可消除视频中代价高昂或不足的时序自注意力操作。在基准数据集上的大量实验验证了我们模块的优越性与泛化性。特别地，所提出的ILA在Kinetics-400数据集上以远低于Swin-L和ViViT-H的计算量取得了88.7%的Top-1准确率。代码已开源至https://github.com/Francis-Rings/ILA。