Access to pre-trained models has recently emerged as a standard across numerous machine learning domains. Unfortunately, access to the original data the models were trained on may not equally be granted. This makes it tremendously challenging to fine-tune, compress models, adapt continually, or to do any other type of data-driven update. We posit that original data access may however not be required. Specifically, we propose Contrastive Abductive Knowledge Extraction (CAKE), a model-agnostic knowledge distillation procedure that mimics deep classifiers without access to the original data. To this end, CAKE generates pairs of noisy synthetic samples and diffuses them contrastively toward a model's decision boundary. We empirically corroborate CAKE's effectiveness using several benchmark datasets and various architectural choices, paving the way for broad application.

翻译：访问预训练模型已成为众多机器学习领域的标准做法。然而，模型训练时所依赖的原始数据可能无法同样被获取。这使得微调、模型压缩、持续适应或任何其他类型的数据驱动更新变得极其困难。我们提出，原始数据访问或许并非必需。具体而言，我们提出对比反绎知识提取（CAKE），这是一种不依赖原始数据即可模拟深度分类器的模型无关知识蒸馏方法。为此，CAKE生成带噪声的合成样本对，并将其对比式地向模型决策边界扩散。我们通过多个基准数据集和多种架构选择实证验证了CAKE的有效性，为其广泛应用铺平了道路。