With the increasing demand for practical applications of Large Language Models (LLMs), many attention-efficient models have been developed to balance performance and computational cost. However, the adversarial robustness of these models remains under-explored. In this work, we design a framework to investigate the trade-off between efficiency, performance, and adversarial robustness of LLMs by comparing three prominent models with varying levels of complexity and efficiency -- Transformer++, Gated Linear Attention (GLA) Transformer, and MatMul-Free LM -- utilizing the GLUE and AdvGLUE datasets. The AdvGLUE dataset extends the GLUE dataset with adversarial samples designed to challenge model robustness. Our results show that while the GLA Transformer and MatMul-Free LM achieve slightly lower accuracy on GLUE tasks, they demonstrate higher efficiency and either superior or comparative robustness on AdvGLUE tasks compared to Transformer++ across different attack levels. These findings highlight the potential of simplified architectures to achieve a compelling balance between efficiency, performance, and adversarial robustness, offering valuable insights for applications where resource constraints and resilience to adversarial attacks are critical.

翻译：随着大型语言模型（LLMs）在实际应用中的需求日益增长，许多注重效率的模型被开发出来，以平衡性能与计算成本。然而，这些模型的对抗鲁棒性仍未得到充分探索。在本研究中，我们设计了一个框架，通过比较三种在复杂性和效率上各不相同的代表性模型——Transformer++、门控线性注意力（GLA）Transformer 和 MatMul-Free LM，并利用 GLUE 和 AdvGLUE 数据集，来探究 LLMs 在效率、性能与对抗鲁棒性之间的权衡。AdvGLUE 数据集在 GLUE 数据集的基础上扩展了旨在挑战模型鲁棒性的对抗样本。我们的结果表明，尽管 GLA Transformer 和 MatMul-Free LM 在 GLUE 任务上的准确率略低，但与 Transformer++ 相比，它们在 AdvGLUE 任务上表现出更高的效率，以及在不同攻击级别下更优或相当的鲁棒性。这些发现凸显了简化架构在实现效率、性能与对抗鲁棒性之间取得引人注目平衡的潜力，为资源受限和对抗攻击韧性至关重要的应用场景提供了有价值的见解。