Online Continual Learning (OCL) studies learning over a continuous data stream without observing any single example more than once, a setting that is closer to the experience of humans and systems that must learn "on-the-wild". Yet, commonly available benchmarks are far from these real-world conditions, because they explicitly signal different tasks, lack latent similarity structure or assume temporal independence between different examples. Here, we propose a new benchmark for OCL based on language modelling in which input alternates between different languages and domains without any explicit delimitation. Additionally, we propose new metrics to study catastrophic forgetting in this setting and evaluate multiple baseline models based on compositions of experts. Finally, we introduce a simple gating technique that learns the latent similarities between different inputs, improving the performance of a Products of Experts model.

翻译：在线持续学习(OCL)研究在连续数据流中学习,而不多次观察任何单一的例子,这种环境更接近于人类和必须学习“在世”的系统的经验。然而,一般的基准与现实世界的条件相去甚远,因为它们明确表明不同的任务,缺乏潜在的相似结构,或假定不同的例子之间具有时间独立性。这里,我们根据语言建模为OCL提出一个新的基准,在语言建模中输入不同语言和领域之间的交替物,而没有明确的划界。此外,我们提出新的衡量标准,以研究在这一设置中灾难性的遗忘,并根据专家的组成对多种基线模型进行评价。最后,我们引入了一种简单的格言技术,学习不同投入之间的潜在相似点,改进专家产品模型的性能。