Modern machine learning pipelines are limited due to data availability, storage quotas, privacy regulations, and expensive annotation processes. These constraints make it difficult or impossible to train and update large-scale models on such dynamic annotated sets. Continual learning directly approaches this problem, with the ultimate goal of devising methods where a deep neural network effectively learns relevant patterns for new (unseen) classes, without significantly altering its performance on previously learned ones. In this paper, we address the problem of continual learning for video data. We introduce PIVOT, a novel method that leverages extensive knowledge in pre-trained models from the image domain, thereby reducing the number of trainable parameters and the associated forgetting. Unlike previous methods, ours is the first approach that effectively uses prompting mechanisms for continual learning without any in-domain pre-training. Our experiments show that PIVOT improves state-of-the-art methods by a significant 27% on the 20-task ActivityNet setup.

翻译：现代机器学习流程受到数据可用性、存储配额、隐私法规以及昂贵标注过程的限制。这些约束使得在动态标注数据集上训练和更新大规模模型变得困难甚至不可能。持续学习直接应对这一问题，其终极目标是设计方法，使深度神经网络能够有效学习新（未见）类别的相关模式，同时不显著改变其在已学习类别上的性能。本文针对视频数据的持续学习问题展开研究。我们提出PIVOT，一种新颖的方法，该方法利用图像域预训练模型中丰富的知识，从而减少可训练参数数量及相关的遗忘问题。与先前方法不同，我们的方法是首个无需域内预训练、有效利用提示机制实现持续学习的方案。实验表明，在20任务ActivityNet设置下，PIVOT将最先进方法的性能显著提升了27%。