As increasingly capable open-weight large language models (LLMs) are deployed, improving their tamper resistance against unsafe modifications, whether accidental or intentional, becomes critical to minimize risks. However, there is no standard approach to evaluate tamper resistance. Varied datasets, metrics, and tampering configurations make it difficult to compare safety, utility, and robustness across different models and defenses. To address this, we introduce TamperBench, the first unified framework to systematically evaluate the tamper resistance of LLMs. TamperBench (i) curates a repository of state-of-the-art weight-space fine-tuning attacks, latent-space representation attacks, and alignment-stage defenses; (ii) enables realistic adversarial evaluation through systematic hyperparameter sweeps per attack-model pair; and (iii) provides both safety and utility evaluations. We use TamperBench to evaluate 21 open-weight LLMs, including defense-augmented variants, across nine tampering threats using standardized safety and capability metrics with hyperparameter sweeps per model-attack pair. The results provide insights including effects of post-training on tamper resistance, that jailbreak-tuning is typically the most severe attack, and that current alignment-stage defenses largely fail to withstand attack sweeps. Code is available at https://github.com/criticalml-uw/TamperBench.

翻译：随着能力日益增强的开放权重大型语言模型（LLMs）被部署，提升其对意外或故意等不安全修改的抗篡改能力成为降低风险的关键。然而，目前尚无评估抗篡改能力的标准方法。不同数据集、指标和篡改配置使得难以比较不同模型和防御之间的安全性、实用性和鲁棒性。为此，我们提出TamperBench——首个系统评估LLMs抗篡改能力的统一框架。TamperBench (i) 整理了一套包含最先进的权重空间微调攻击、潜在空间表征攻击和对齐阶段防御的基准库；(ii) 通过针对每对攻击-模型进行系统超参数扫描，实现逼真的对抗性评估；(iii) 同时提供安全性和实用性评估。我们使用TamperBench对21个开放权重LLMs（包括增强防御的变体）进行了评估，涵盖9种篡改威胁，采用标准化安全性与能力指标，并对每对模型-攻击组合进行超参数扫描。研究结果揭示了训练后处理对抗篡改能力的影响，表明越狱微调通常是最严重的攻击，且当前对齐阶段防御基本无法抵御攻击扫描。代码见https://github.com/criticalml-uw/TamperBench。