We present PFP, a neural network structure to compress long videos into short contexts, with an explicit pretraining objective to preserve the high-frequency details of single frames at arbitrary temporal positions. The baseline model can compress a 20-second video into a context at about 5k length, where random frames can be retrieved with perceptually preserved appearances. Such pretrained models can be directly fine-tuned as memory encoders for autoregressive video models, enabling long history memory with low context cost and relatively low fidelity loss. We evaluate the framework with ablative settings and discuss the trade-offs of possible neural architecture designs.

翻译：我们提出PFP，一种将长视频压缩为短上下文的神经网络结构，其明确的预训练目标是在任意时间位置保持单帧的高频细节。基线模型可将20秒视频压缩至约5k长度的上下文，其中随机帧能够以感知保留的外观被检索。此类预训练模型可直接微调为自回归视频模型的记忆编码器，实现以低上下文成本和相对低保真度损失的长历史记忆。我们通过消融实验评估该框架，并探讨可能的神经架构设计中的权衡关系。