Model stealing aims at inferring a victim model's functionality at a fraction of the original training cost. While the goal is clear, in practice the model's architecture, weight dimension, and original training data can not be determined exactly, leading to mutual uncertainty during stealing. In this work, we explicitly tackle this uncertainty by generating multiple possible networks and combining their predictions to improve the quality of the stolen model. For this, we compare five popular uncertainty quantification models in a model stealing task. Surprisingly, our results indicate that the considered models only lead to marginal improvements in terms of label agreement (i.e., fidelity) to the stolen model. To find the cause of this, we inspect the diversity of the model's prediction by looking at the prediction variance as a function of training iterations. We realize that during training, the models tend to have similar predictions, indicating that the network diversity we wanted to leverage using uncertainty quantification models is not (high) enough for improvements on the model stealing task.

翻译：模型窃取旨在以原始训练成本的一小部分来推断受害者模型的功能。虽然目标明确，但在实践中，模型的架构、权重维度和原始训练数据无法精确确定，导致窃取过程中存在相互不确定性。在本工作中，我们通过生成多个可能的网络并组合其预测来显式处理这种不确定性，以提高窃取模型的质量。为此，我们在模型窃取任务中比较了五种流行的不确定性量化模型。令人惊讶的是，我们的结果表明，所考虑的模型在被窃取模型的标签一致性（即保真度）方面仅带来微小改进。为探究其原因，我们通过检查预测方差随训练迭代次数的变化来审视模型预测的多样性。我们发现，在训练过程中，模型倾向于产生相似的预测，这表明我们希望通过不确定性量化模型利用的网络多样性对于改进模型窃取任务而言尚不足够。