In this paper, we make the first attempt to understand and test potential computation efficiency robustness in state-of-the-art LLMs. By analyzing the working mechanism and implementation of 20,543 public-accessible LLMs, we observe a fundamental property in LLMs that could be manipulated in an adversarial manner to reduce computation efficiency significantly. Our key motivation is to generate test inputs that could sufficiently delay the generation of EOS such that LLMs would have to go through enough iterations to satisfy the pre-configured threshold. We present \tool, which can work under both white-box setting and black-box setting. In the white-box scenario, \tool develops a gradient-guided technique that searches for a minimal and unnoticeable perturbation at character-level, token-level, and structure-level. In the black-box scenario, \tool employs a causal inference-based approach to find critical tokens and similarly applies three levels of imperceptible perturbation to them. Both the white-box and black-box settings effectively delay the appearance of EOS, compelling these inputs to reach the naturally-unreachable threshold. To demonstrate the effectiveness of \tool, we conduct a systematic evaluation on nine public-available LLMs: Google T5, AllenAI WMT14, Helsinki-NLP translator, Facebook FairSeq, UNICAMP-DL translator, MarianMT, Google FLAN-T5, MBZUAI LaMini-GPT and Salesforce CodeGen. Experimental results show that \tool can increase on average LLMs' response latency and energy consumption by 325\% to 3244\% and 344\% to 3616\%, respectively, by perturbing just one character or token in the input sentence.

翻译：本文首次尝试理解并测试前沿大型语言模型（LLM）潜在的计算效率鲁棒性问题。通过分析20,543个公开可访问的LLM的工作机制与实现方式，我们发现LLM中存在一个基础特性，该特性可能被以对抗性方式操纵，从而显著降低计算效率。我们的核心动机是生成能够充分延迟EOS（结束符）生成的测试输入，迫使LLM必须经过足够迭代次数才能达到预设阈值。我们提出了\tool，该工具可在白盒与黑盒两种设置下工作。在白盒场景中，\tool开发了一种梯度引导技术，可在字符级、词元级和结构级搜索最小化且不易察觉的扰动。在黑盒场景中，\tool采用基于因果推断的方法定位关键词元，并同样施加三个层级的不可感知扰动。白盒与黑盒设置均能有效延迟EOS的出现，迫使这些输入达到自然情况下无法触及的阈值。为验证\tool的有效性，我们对九个公开可用的LLM进行了系统评估：Google T5、AllenAI WMT14、Helsinki-NLP translator、Facebook FairSeq、UNICAMP-DL translator、MarianMT、Google FLAN-T5、MBZUAI LaMini-GPT以及Salesforce CodeGen。实验结果表明，仅需对输入句子中的一个字符或词元施加扰动，\tool平均可将LLM的响应延迟和能耗分别提升325%至3244%与344%至3616%。