We propose RanDumb to examine the efficacy of continual representation learning. RanDumb embeds raw pixels using a fixed random transform which approximates an RBF-Kernel, initialized before seeing any data, and learns a simple linear classifier on top. We present a surprising and consistent finding: RanDumb significantly outperforms the continually learned representations using deep networks across numerous continual learning benchmarks, demonstrating the poor performance of representation learning in these scenarios. RanDumb stores no exemplars and performs a single pass over the data, processing one sample at a time. It complements GDumb, operating in a low-exemplar regime where GDumb has especially poor performance. We reach the same consistent conclusions when RanDumb is extended to scenarios with pretrained models replacing the random transform with pretrained feature extractor. Our investigation is both surprising and alarming as it questions our understanding of how to effectively design and train models that require efficient continual representation learning, and necessitates a principled reinvestigation of the widely explored problem formulation itself. Our code is available at https://github.com/drimpossible/RanDumb.

翻译：我们提出RanDumb方法以检验持续表示学习的效果。RanDumb采用固定随机变换对原始像素进行嵌入，该变换近似于径向基核函数（RBF-Kernel），且初始化于任何数据观测之前，并在其上学习一个简单的线性分类器。我们呈现一个令人惊讶且一致的发现：在众多持续学习基准测试中，RanDumb显著优于使用深度网络持续学习的表示方法，揭示了表示学习在这些场景中的低效表现。RanDumb无需存储任何样本，仅对数据进行单遍处理，每次仅处理一个样本。它补充了GDumb方法，在GDumb表现尤为不佳的低样本记忆场景中发挥作用。当我们将RanDumb扩展至使用预训练模型替换随机变换为预训练特征提取器的场景时，得出了同样一致的结论。这一发现令人惊讶且警醒，因为它挑战了我们对如何有效设计并训练需要高效持续表示学习的模型的理解，并迫使我们对广泛探索的问题本身进行原则性的重新审视。我们的代码已开源至 https://github.com/drimpossible/RanDumb。