As an emerging type of Neural Networks (NNs), Transformers are used in many domains ranging from Natural Language Processing to Autonomous Driving. In this paper, we study the robustness problem of Transformers, a key characteristic as low robustness may cause safety concerns. Specifically, we focus on Sparsemax-based Transformers and reduce the finding of their maximum robustness to a Mixed Integer Quadratically Constrained Programming (MIQCP) problem. We also design two pre-processing heuristics that can be embedded in the MIQCP encoding and substantially accelerate its solving. We then conduct experiments using the application of Land Departure Warning to compare the robustness of Sparsemax-based Transformers against that of the more conventional Multi-Layer-Perceptron (MLP) NNs. To our surprise, Transformers are not necessarily more robust, leading to profound considerations in selecting appropriate NN architectures for safety-critical domain applications.

翻译：作为新兴的神经网络类型，Transformer已被广泛应用于从自然语言处理到自动驾驶等多个领域。本文针对Transformer的鲁棒性问题展开研究——该特性至关重要，因为低鲁棒性可能引发安全隐患。具体而言，我们聚焦于基于Sparsemax的Transformer，将其最大鲁棒性求解问题转化为混合整数二次约束规划问题。同时设计两种可嵌入MIQCP编码的预处理启发式策略，显著提升求解效率。通过车道偏离预警应用场景的实验，我们对比了基于Sparsemax的Transformer与传统多层感知器神经网络的鲁棒性。令人意外的是，Transformer未必具有更优的鲁棒性，这一发现对在安全关键领域应用中如何选择恰当的神经网络架构具有重要启示。