In this paper, we propose a new Deep Neural Network (DNN) testing algorithm called the Constrained Gradient Descent (CGD) method, and an implementation we call CGDTest aimed at exposing security and robustness issues such as adversarial robustness and bias in DNNs. Our CGD algorithm is a gradient-descent (GD) method, with the twist that the user can also specify logical properties that characterize the kinds of inputs that the user may want. This functionality sets CGDTest apart from other similar DNN testing tools since it allows users to specify logical constraints to test DNNs not only for $\ell_p$ ball-based adversarial robustness but, more importantly, includes richer properties such as disguised and flow adversarial constraints, as well as adversarial robustness in the NLP domain. We showcase the utility and power of CGDTest via extensive experimentation in the context of vision and NLP domains, comparing against 32 state-of-the-art methods over these diverse domains. Our results indicate that CGDTest outperforms state-of-the-art testing tools for $\ell_p$ ball-based adversarial robustness, and is significantly superior in testing for other adversarial robustness, with improvements in PAR2 scores of over 1500% in some cases over the next best tool. Our evaluation shows that our CGD method outperforms competing methods we compared against in terms of expressibility (i.e., a rich constraint language and concomitant tool support to express a wide variety of properties), scalability (i.e., can be applied to very large real-world models with up to 138 million parameters), and generality (i.e., can be used to test a plethora of model architectures).

翻译：本文提出了一种新的深度神经网络（DNN）测试算法，称为约束梯度下降（CGD）方法，并实现了相应的工具CGDTest，旨在暴露DNN中的安全性与鲁棒性问题，例如对抗鲁棒性和偏差。我们的CGD算法是一种梯度下降（GD）方法，其独特之处在于用户还可以指定逻辑属性，以刻画可能需要的输入类型。这一功能使CGDTest有别于其他类似的DNN测试工具，因为它允许用户指定逻辑约束，不仅用于测试基于ℓ_p球的对抗鲁棒性，更重要的是，还能涵盖更丰富的属性，如伪装和流式对抗约束，以及自然语言处理领域的对抗鲁棒性。我们通过在视觉和NLP领域的大量实验，与32种最先进的方法进行对比，展示了CGDTest的实用性和强大功能。结果表明，CGDTest在基于ℓ_p球的对抗鲁棒性测试中优于现有最先进的测试工具，并且在其他对抗鲁棒性测试中表现显著优越，其中在某些情况下，PAR2分数相比次优工具提升了超过1500%。我们的评估表明，CGD方法在表达能力（即丰富的约束语言及配套工具支持以表达多种属性）、可扩展性（即可应用于参数高达1.38亿的超大型真实世界模型）和通用性（即可用于测试多种模型架构）方面均优于所对比的竞争方法。