Contrastive language-image pretraining (CLIP) has demonstrated remarkable success in various image tasks. However, how to extend CLIP with effective temporal modeling is still an open and crucial problem. Existing factorized or joint spatial-temporal modeling trades off between the efficiency and performance. While modeling temporal information within straight through tube is widely adopted in literature, we find that simple frame alignment already provides enough essence without temporal attention. To this end, in this paper, we proposed a novel Implicit Learnable Alignment (ILA) method, which minimizes the temporal modeling effort while achieving incredibly high performance. Specifically, for a frame pair, an interactive point is predicted in each frame, serving as a mutual information rich region. By enhancing the features around the interactive point, two frames are implicitly aligned. The aligned features are then pooled into a single token, which is leveraged in the subsequent spatial self-attention. Our method allows eliminating the costly or insufficient temporal self-attention in video. Extensive experiments on benchmarks demonstrate the superiority and generality of our module. Particularly, the proposed ILA achieves a top-1 accuracy of 88.7% on Kinetics-400 with much fewer FLOPs compared with Swin-L and ViViT-H. Code is released at https://github.com/Francis-Rings/ILA .

翻译：对比语言-图像预训练（CLIP）已在多种图像任务中表现出显著成功。然而，如何将CLIP扩展至有效的时序建模仍是一个开放且关键的问题。现有分解式或联合时空建模在效率与性能之间进行权衡。尽管沿直通管道建模时序信息在文献中被广泛采用，我们发现简单的帧对齐无需时序注意力已能提供足够本质。为此，本文提出一种新颖的隐式可学习对齐（ILA）方法，在实现极高性能的同时最小化时序建模工作量。具体而言，对于一组帧对，我们在每帧中预测一个交互点，将其作为互信息丰富区域。通过增强交互点周围的特征，两帧被隐式对齐。对齐后的特征随后池化为单一token，用于后续的空间自注意力。该方法使我们能够消除视频中昂贵或不充分的时序自注意力。在多个基准数据集上的大量实验证明了本模块的优越性与通用性。值得注意的是，所提出的ILA方法在Kinetics-400上以远低于Swin-L和ViViT-H的计算量取得了88.7%的top-1准确率。代码已开源至https://github.com/Francis-Rings/ILA。