How does the choice of training data influence an AI model? This broad question is of central importance to interpretability, privacy, and basic science. At its technical core is the data deletion problem: after a reasonable amount of precomputation, quickly predict how the model would behave in a given situation if a given subset of training data had been excluded from the learning algorithm. We present a data deletion scheme capable of predicting model outputs with vanishing error $\varepsilon$ and failure probability $δ$ in the deep learning setting. Our precomputation and prediction algorithms are only $\tilde{O}(\log(1/δ)/\varepsilon^2)$ factors slower than regular training and inference, respectively. The storage requirements are those of $\tilde{O}(\log(1/δ)/\varepsilon^2)$ models. Our proof is based on an assumption that we call stability. In contrast to the assumptions made by prior work, stability appears to be fully compatible with learning powerful AI models. In support of this, we show that stability is satisfied in a minimal set of experiments with microgpt. Our code is available at https://github.com/SamSpo1/microgpt-sketch. At a technical level, our work is based on a new method for locally sketching an arithmetic circuit by computing higher-order derivatives in random complex directions. Forward-mode automatic differentiation allows cheap computation of these derivatives.

翻译：训练数据的选择如何影响人工智能模型？这一宽泛问题对于可解释性、隐私保护和基础科学具有核心重要性。其技术核心是数据删除问题：在完成适度量的预计算后，快速预测若学习算法中排除了给定训练数据子集，模型在特定情形下将如何表现。我们提出一种数据删除方案，能够在深度学习环境下以可忽略误差$\varepsilon$和失败概率$\delta$预测模型输出。我们的预计算和预测算法分别仅比常规训练和推理慢$\tilde{O}(\log(1/\delta)/\varepsilon^2)$倍，存储需求则为$\tilde{O}(\log(1/\delta)/\varepsilon^2)$个模型的规模。我们的证明基于一个称为“稳定性”的假设。与先前工作所作的假设不同，稳定性似乎与学习强大AI模型完全兼容。为支持这一观点，我们在microgpt的最小实验集合中验证了稳定性满足条件。我们的代码可在https://github.com/SamSpo1/microgpt-sketch获取。在技术层面，本工作基于一种通过计算随机复数方向高阶导数来局部勾勒算术电路的新方法，而前向模式自动微分可实现这些导数的廉价计算。