Recent studies have made notable progress in video representation learning by transferring image-pretrained models to video tasks, typically with complex temporal modules and video fine-tuning. However, fine-tuning heavy modules may compromise inter-video semantic separability, i.e., the essential ability to distinguish objects across videos. While reducing the tunable parameters hinders their intra-video temporal consistency, which is required for stable representations of the same object within a video. This dilemma indicates a potential trade-off between the intra-video temporal consistency and inter-video semantic separability during image-to-video transfer. To this end, we propose the Consistency-Separability Trade-off Transfer Learning (Co-Settle) framework, which applies a lightweight projection layer on top of the frozen image-pretrained encoder to adjust representation space with a temporal cycle consistency objective and a semantic separability constraint. We further provide a theoretical support showing that the optimized projection yields a better trade-off between the two properties under appropriate conditions. Experiments on eight image-pretrained models demonstrate consistent improvements across multiple levels of video tasks with only five epochs of self-supervised training. The code is available at https://github.com/yafeng19/Co-Settle.

翻译：近期研究通过将图像预训练模型迁移至视频任务，在视频表征学习领域取得了显著进展，通常采用复杂的时间模块和视频微调方法。然而，微调大量模块可能损害视频间语义可分离性——即跨视频区分物体的核心能力；而减少可调参数又会阻碍视频内时间一致性——这对同一物体在视频中稳定表征至关重要。这一困境揭示了图像到视频迁移过程中视频内时间一致性与视频间语义可分离性之间潜在的权衡关系。为此，我们提出一致性-可分离性权衡迁移学习框架（Co-Settle），在冻结的图像预训练编码器顶部添加轻量级投影层，通过时间循环一致性目标和语义可分离性约束调整表征空间。我们进一步提供理论支撑，证明在适当条件下优化投影能在两种特性间实现更优权衡。在八个图像预训练模型上的实验表明，仅需五轮自监督训练，即可在多层次视频任务中取得一致性提升。代码已开源至https://github.com/yafeng19/Co-Settle。