While advanced machine learning (ML) models are deployed in numerous real-world applications, previous works demonstrate these models have security and privacy vulnerabilities. Various empirical research has been done in this field. However, most of the experiments are performed on target ML models trained by the security researchers themselves. Due to the high computational resource requirement for training advanced models with complex architectures, researchers generally choose to train a few target models using relatively simple architectures on typical experiment datasets. We argue that to understand ML models' vulnerabilities comprehensively, experiments should be performed on a large set of models trained with various purposes (not just the purpose of evaluating ML attacks and defenses). To this end, we propose using publicly available models with weights from the Internet (public models) for evaluating attacks and defenses on ML models. We establish a database, namely SecurityNet, containing 910 annotated image classification models. We then analyze the effectiveness of several representative attacks/defenses, including model stealing attacks, membership inference attacks, and backdoor detection on these public models. Our evaluation empirically shows the performance of these attacks/defenses can vary significantly on public models compared to self-trained models. We share SecurityNet with the research community. and advocate researchers to perform experiments on public models to better demonstrate their proposed methods' effectiveness in the future.

翻译：尽管先进的机器学习（ML）模型已部署在众多现实应用中，但先前研究表明这些模型存在安全与隐私漏洞。该领域已有大量实证研究，然而多数实验是在安全研究人员自行训练的目标ML模型上进行的。由于训练具有复杂架构的先进模型需要极高的计算资源，研究人员通常选择在典型实验数据集上使用相对简单的架构训练少量目标模型。我们认为，要全面理解ML模型的脆弱性，实验应基于大量具有不同训练目标（而不仅是评估ML攻击与防御）的模型进行。为此，我们提议使用来自互联网的公开可用模型及其权重（公开模型）来评估ML模型的攻击与防御手段。我们构建了名为SecurityNet的数据库，包含910个带标注的图像分类模型。随后，我们分析了数种代表性攻击/防御方法（包括模型窃取攻击、成员推断攻击和后门检测）在这些公开模型上的有效性。实验结果实证表明，与自训练模型相比，这些攻击/防御方法在公开模型上的表现可能存在显著差异。我们将SecurityNet共享给研究社区，并倡导未来研究人员在公开模型上进行实验，以更充分地验证其提出方法的有效性。