We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 30 out of 33 video understanding benchmarks.

翻译：我们提出VideoPrism，这是一种通用视频编码器，能够通过单个冻结模型处理多种视频理解任务。我们在包含3600万高质量视频-文本对和5.82亿带有噪声平行文本（如自动语音识别转录）的视频片段组成的异构语料库上预训练了VideoPrism。该预训练方法通过全局-局部语义视频嵌入蒸馏和令牌洗牌机制改进了掩码自编码，使VideoPrism能够主要聚焦于视频模态，同时利用与视频相关的宝贵文本。我们系统性地在四大类视频理解任务（从网络视频问答到科学领域的计算机视觉应用）上测试了VideoPrism，在33个视频理解基准中的30个上取得了领先性能。