Access to pre-trained models has recently emerged as a standard across numerous machine learning domains. Unfortunately, access to the original data the models were trained on may not equally be granted. This makes it tremendously challenging to fine-tune, compress models, adapt continually, or to do any other type of data-driven update. We posit that original data access may however not be required. Specifically, we propose Contrastive Abductive Knowledge Extraction (CAKE), a model-agnostic knowledge distillation procedure that mimics deep classifiers without access to the original data. To this end, CAKE generates pairs of noisy synthetic samples and diffuses them contrastively toward a model's decision boundary. We empirically corroborate CAKE's effectiveness using several benchmark datasets and various architectural choices, paving the way for broad application.

翻译：对预训练模型的访问已成为众多机器学习领域的标准做法。然而，模型训练所使用的原始数据可能无法同样获得授权。这使得微调、压缩模型、持续适应或执行任何其他类型的数据驱动更新面临巨大挑战。我们认为，原始数据访问可能并非必要。具体而言，我们提出对比反演知识提取（CAKE），这是一种无需访问原始数据即可模仿深度分类器的模型无关知识蒸馏方法。为此，CAKE生成带噪声的合成样本对，并将其对比式地扩散至模型的决策边界。我们通过多个基准数据集及不同架构选择实证验证了CAKE的有效性，为其广泛应用铺平了道路。