Machine unlearning facilitates personal data ownership, including the ``right to be forgotten''. The proliferation of applications of \emph{neural networks} (NNs) trained on users' personal data calls for the need to develop algorithms to unlearn an NN. Since retraining is costly, efficiency is often achieved through approximate unlearning which aims to unlearn a trained NN to be close to the retrained one (in distribution). Though the Newton's method has been used by previous works to approximately unlearn linear models, adapting it for unlearning an NN often encounters degenerate Hessians that make computing the Newton's update impossible. In this paper, we will first show that when coupled with naive yet often effective solutions to mitigate the degeneracy issue for unlearning, the Newton's method surprisingly suffers from catastrophic forgetting. To overcome this difficulty, we revise the Newton's method to include a theoretically justified regularizer and propose a cubic-regularized Newton's method for unlearning an NN. The cubic regularizer comes with the benefits of not requiring manual finetuning and affording a natural interpretation. Empirical evaluation on several models and real-world datasets shows that our method is more resilient to catastrophic forgetting and performs better than the baselines, especially in sequential unlearning.

翻译：机器遗忘技术有助于实现个人数据所有权，包括“被遗忘权”。基于用户个人数据训练的神经网络应用日益广泛，亟需开发针对神经网络的遗忘算法。由于重新训练成本高昂，通常通过近似遗忘来实现效率提升，其目标是将已训练神经网络遗忘至与重新训练网络（在分布上）相近的状态。尽管先前研究已采用牛顿方法实现线性模型的近似遗忘，但将其应用于神经网络遗忘时常会遇到海森矩阵退化问题，导致无法计算牛顿更新量。本文首先证明，当结合朴素但通常有效的退化缓解方案进行遗忘时，牛顿方法会出现灾难性遗忘现象。为克服此难题，我们改进牛顿方法，引入具有理论依据的正则化项，提出用于神经网络遗忘的三次正则化牛顿方法。该三次正则化器具有无需手动调参和便于自然解释的优势。在多个模型和真实数据集上的实验评估表明，我们的方法对灾难性遗忘具有更强鲁棒性，在序列遗忘场景中尤其优于基线方法。