An obstacle to artificial general intelligence is set by the continual learning of multiple tasks of different nature. Recently, various heuristic tricks, both from machine learning and from neuroscience angles, were proposed, but they lack a unified theory ground. Here, we focus on the continual learning in single-layered and multi-layered neural networks of binary weights. A variational Bayesian learning setting is thus proposed, where the neural network is trained in a field-space, rather than the gradient-ill-defined discrete-weight space, and furthermore, the weight uncertainty is naturally incorporated, and modulates the synaptic resources among tasks. From a physics perspective, we translate the variational continual learning into the Franz-Parisi thermodynamic potential framework, where the previous task knowledge acts as a prior and a reference as well. We thus interprete the continual learning of the binary perceptron in a teacher-student setting as a Franz-Parisi potential computation. The learning performance can then be analytically studied with mean-field order parameters, whose predictions coincide with the numerical experiments using stochastic gradient descent methods. Based on the variational principle and Gaussian field approximation of internal preactivations in hidden layers, we also derive the learning algorithm considering weight uncertainty, which outperforms the current metaplasticity algorithm in continually learning multiple tasks. Our proposed principled frameworks also connect to elastic weight consolidation, and neuroscience inspired metaplasticity, providing a theory-grounded method for the real-world multi-task learning with deep networks.

翻译：通用人工智能的一大障碍在于如何处理多种不同性质的持续学习任务。近年来，从机器学习和神经科学角度提出了各种启发式方法，但这些方法缺乏统一的理论基础。本文聚焦于二进制权重的单层与多层神经网络中的持续学习问题。我们提出了一种变分贝叶斯学习框架，其中神经网络在场空间中训练，而非梯度定义不明确的离散权重空间，并且自然融合了权重不确定性，从而在各任务间调节突触资源。从物理学视角，我们将变分持续学习转化为Franz-Parisi热力学势框架，其中先前任务知识既作为先验也作为参考。因此，在教师-学生设置下，我们将二值感知机的持续学习解释为Franz-Parisi势的计算问题。通过平均场序参数，可解析研究学习性能，其预测结果与使用随机梯度下降法的数值实验相吻合。基于变分原理与隐藏层内部预激活的高斯场近似，我们进一步推导了考虑权重不确定性的学习算法，该算法在持续学习多任务时优于当前元可塑性算法。本文提出的原则性框架也与弹性权重巩固及神经科学启发的元可塑性相关联，为基于深度网络的实际多任务学习提供了理论支持方法。