Distillation is the task of replacing a complicated machine learning model with a simpler model that approximates the original [BCNM06,HVD15]. Despite many practical applications, basic questions about the extent to which models can be distilled, and the runtime and amount of data needed to distill, remain largely open. To study these questions, we initiate a general theory of distillation, defining PAC-distillation in an analogous way to PAC-learning [Val84]. As applications of this theory: (1) we propose new algorithms to extract the knowledge stored in the trained weights of neural networks -- we show how to efficiently distill neural networks into succinct, explicit decision tree representations when possible by using the ``linear representation hypothesis''; and (2) we prove that distillation can be much cheaper than learning from scratch, and make progress on characterizing its complexity.

翻译：蒸馏是将复杂的机器学习模型替换为近似原模型的更简单模型的任务[BCNM06,HVD15]。尽管有许多实际应用，但关于模型可蒸馏程度、蒸馏所需运行时间和数据量的基本问题仍然悬而未决。为研究这些问题，我们开创了蒸馏的一般理论，以类似于PAC学习的方式定义了PAC蒸馏[Val84]。作为该理论的应用：（1）我们提出了新算法来提取存储在神经网络训练权重中的知识——展示了如何利用"线性表示假设"在可能的情况下将神经网络高效蒸馏为简洁、显式的决策树表示；（2）我们证明了蒸馏可能比从头学习更高效，并对其复杂度刻画取得了进展。