How does the choice of training data influence an AI model? This question is of central importance to interpretability, privacy, and basic science. At its core is the data deletion problem: after a reasonable amount of precomputation, quickly predict how the model would behave in a given situation if a given subset of training data had been excluded from the learning algorithm. We present a data deletion scheme capable of predicting model outputs with vanishing error $\varepsilon$ in the deep learning setting. Our precomputation and prediction algorithms are only $\mathrm{poly}(1/\varepsilon)$ factors slower than regular training and inference, respectively. The storage requirements are those of $\mathrm{poly}(1/\varepsilon)$ models. Our proof is based on an assumption that we call "stability." In contrast to the assumptions made by prior work, stability appears to be fully compatible with learning powerful AI models. In support of this, we show that stability is satisfied in a minimal set of experiments with microgpt. Our code is available at https://github.com/SamSpo1/microgpt-sketch. At a technical level, our work is based on a new method for locally sketching an arithmetic circuit by computing higher-order derivatives in random complex directions. Forward-mode automatic differentiation allows cheap computation of these derivatives.

翻译：训练数据的选择如何影响人工智能模型？这一问题对于可解释性、隐私保护和基础科学至关重要。其核心是数据删除问题：在合理预计算后，快速预测若从学习算法中排除给定训练数据子集，模型在特定情境下的行为将如何变化。我们提出一种数据删除方案，能够在深度学习场景下以消失误差ε预测模型输出。我们的预计算和预测算法分别仅比常规训练和推理慢poly(1/ε)倍，存储需求相当于poly(1/ε)个模型的规模。我们的证明基于称为“稳定性”的假设。与先前工作所采用的假设不同，稳定性似乎与学习强大的人工智能模型完全兼容。为支持这一观点，我们通过微GPT的最小实验集验证了稳定性的满足。我们的代码公开于https://github.com/SamSpo1/microgpt-sketch。在技术层面，我们的工作基于一种新型方法：通过计算随机复方向上的高阶导数，局部勾画算术电路。前向模式自动微分使得这些导数的计算成本低廉。