Video-and-language understanding has a variety of applications in the industry, such as video question answering, text-video retrieval and multi-label classification. Existing video-and-language understanding methods generally adopt heavy multi-modal encoders and feature fusion modules, which consume large amounts of GPU memory. Especially, they have difficulty dealing with dense video frames or long text that are prevalent in industrial applications. In this paper, we propose MuLTI, a highly accurate and memory-efficient video-and-language understanding model that achieves efficient and effective feature fusion through feature sampling and attention modules. Therefore, MuLTI can handle longer sequences with limited GPU memory. Then, we introduce an attention-based adapter to the encoders, which finetunes the shallow features to improve the model's performance with low GPU memory consumption. Finally, to further improve the model's performance, we introduce a new pretraining task named Multiple Choice Modeling to bridge the task gap between pretraining and downstream tasks and enhance the model's ability to align the video and the text. Benefiting from the efficient feature fusion module, the attention-based adapter and the new pretraining task, MuLTI achieves state-of-the-art performance on multiple datasets. Implementation and pretrained models will be released.

翻译：视频-语言理解在工业界具有广泛的应用，例如视频问答、文本-视频检索和多标签分类。现有视频-语言理解方法通常采用重型多模态编码器和特征融合模块，消耗大量GPU内存。特别是，它们难以处理工业应用中普遍存在的密集视频帧或长文本。本文提出MuLTI，一种高精度且内存高效的视频-语言理解模型，通过特征采样和注意力模块实现高效有效的特征融合。因此，MuLTI能够在有限GPU内存下处理更长的序列。此外，我们引入基于注意力的适配器到编码器中，通过微调浅层特征来提升模型性能，同时保持低GPU内存消耗。最后，为进一步提升模型性能，我们提出一项名为多重选择建模（Multiple Choice Modeling）的新预训练任务，旨在弥合预训练与下游任务之间的差距，并增强模型对齐视频与文本的能力。得益于高效特征融合模块、基于注意力的适配器以及新的预训练任务，MuLTI在多个数据集上达到了最先进性能。实现代码与预训练模型将公开发布。