Distillation is the task of replacing a complicated machine learning model with a simpler model that approximates the original [BCNM06,HVD15]. Despite many practical applications, basic questions about the extent to which models can be distilled, and the runtime and amount of data needed to distill, remain largely open. To study these questions, we initiate a general theory of distillation, defining PAC-distillation in an analogous way to PAC-learning [Val84]. As applications of this theory: (1) we propose new algorithms to extract the knowledge stored in the trained weights of neural networks -- we show how to efficiently distill neural networks into succinct, explicit decision tree representations when possible by using the ``linear representation hypothesis''; and (2) we prove that distillation can be much cheaper than learning from scratch, and make progress on characterizing its complexity.

翻译：蒸馏是将复杂的机器学习模型替换为近似原模型的更简单模型的任务[BCNM06,HVD15]。尽管有许多实际应用，但关于模型可蒸馏的程度、蒸馏所需的运行时间和数据量等基本问题在很大程度上仍未解决。为研究这些问题，我们首次提出了蒸馏的一般理论，以类似于PAC学习[Val84]的方式定义了PAC-蒸馏。作为该理论的应用：(1) 我们提出了新算法来提取神经网络训练权重中存储的知识——展示了如何利用“线性表示假设”，在可能的情况下高效地将神经网络蒸馏为简洁、显式的决策树表示；(2) 我们证明了蒸馏可能比从头学习成本更低，并在刻画其复杂性方面取得了进展。