Attention Networks (ATNs) such as Transformers are used in many domains ranging from Natural Language Processing to Autonomous Driving. In this paper, we study the robustness problem of ATNs, a key characteristic where low robustness may cause safety concerns. Specifically, we focus on Sparsemax-based ATNs and reduce the finding of their maximum robustness to a Mixed Integer Quadratically Constrained Programming (MIQCP) problem. We also design two pre-processing heuristics that can be embedded in the MIQCP encoding and substantially accelerate its solving. We then conduct experiments using the application of Land Departure Warning to compare the robustness of Sparsemax-based ATNs against that of the more conventional Multi-Layer-Perceptron (MLP) Neural Networks (NNs). To our surprise, ATNs are not necessarily more robust, leading to profound considerations in selecting appropriate NN architectures for safety-critical domain applications.

翻译：注意力网络（如Transformer）已广泛应用于从自然语言处理到自动驾驶的诸多领域。本文研究注意力网络的鲁棒性问题——该关键特性若水平不足可能引发安全隐患。具体而言，我们聚焦基于Sparsemax的注意力网络，将其最大鲁棒性求解问题转化为混合整数二次约束规划（MIQCP）问题。同时设计两种可嵌入MIQCP编码的预处理启发式算法，显著提升求解效率。通过车道偏离预警应用场景的实验，我们对比了基于Sparsemax的注意力网络与传统多层感知器（MLP）神经网络的鲁棒性。令人意外的是，注意力网络未必更具鲁棒性，这一发现为安全关键领域应用中神经网络架构的合理选择提供了深刻思考。