Online learning of deep neural networks suffers from challenges such as hysteretic non-incremental updating, increasing memory usage, past retrospective retraining, and catastrophic forgetting. To alleviate these drawbacks and achieve progressive immediate decision-making, we propose a novel Incremental Online Learning (IOL) process of Randomized Neural Networks (Randomized NN), a framework facilitating continuous improvements to Randomized NN performance in restrictive online scenarios. Within the framework, we further introduce IOL with ridge regularization (-R) and IOL with forward regularization (-F). -R generates stepwise incremental updates without retrospective retraining and avoids catastrophic forgetting. Moreover, we substituted -R with -F as it enhanced precognition learning ability using semi-supervision and realized better online regrets to offline global experts compared to -R during IOL. The algorithms of IOL for Randomized NN with -R/-F on non-stationary batch stream were derived respectively, featuring recursive weight updates and variable learning rates. Additionally, we conducted a detailed analysis and theoretically derived relative cumulative regret bounds of the Randomized NN learners with -R/-F in IOL under adversarial assumptions using a novel methodology and presented several corollaries, from which we observed the superiority on online learning acceleration and regret bounds of employing -F in IOL. Finally, our proposed methods were rigorously examined across regression and classification tasks on diverse datasets, which distinctly validated the efficacy of IOL frameworks of Randomized NN and the advantages of forward regularization.

翻译：深度神经网络的在线学习面临诸多挑战，如迟滞的非增量更新、内存使用量增长、过往数据回顾式重训练以及灾难性遗忘。为缓解这些缺陷并实现渐进式即时决策，我们提出了一种新颖的随机神经网络增量在线学习框架，该框架能够在受限的在线场景中持续提升随机神经网络的性能。在此框架内，我们进一步引入了带岭正则化的增量在线学习与带前向正则化的增量在线学习。-R 方法实现了无需回顾式重训练的逐步增量更新，并避免了灾难性遗忘。此外，我们以 -F 替代 -R，因其通过半监督学习增强了预认知学习能力，并在增量在线学习过程中相比 -R 实现了相对于离线全局专家更优的在线遗憾度。我们分别推导了针对非平稳批量数据流的、采用 -R/-F 的随机神经网络增量在线学习算法，其特点在于递归的权重更新与可变学习率。此外，我们进行了详细分析，并采用一种新颖方法在对抗性假设下从理论上推导了增量在线学习中采用 -R/-F 的随机神经网络学习器的相对累积遗憾界，同时提出了若干推论。从中我们观察到，在增量在线学习中采用 -F 在在线学习加速和遗憾界方面具有优越性。最后，我们在多种数据集的回归和分类任务上对所提方法进行了严格检验，结果明确验证了随机神经网络增量在线学习框架的有效性以及前向正则化的优势。