Generative Pre-trained Transformer (GPT) models have exhibited exciting progress in their capabilities, capturing the interest of practitioners and the public alike. Yet, while the literature on the trustworthiness of GPT models remains limited, practitioners have proposed employing capable GPT models for sensitive applications such as healthcare and finance -- where mistakes can be costly. To this end, this work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5, considering diverse perspectives -- including toxicity, stereotype bias, adversarial robustness, out-of-distribution robustness, robustness on adversarial demonstrations, privacy, machine ethics, and fairness. Based on our evaluations, we discover previously unpublished vulnerabilities to trustworthiness threats. For instance, we find that GPT models can be easily misled to generate toxic and biased outputs and leak private information in both training data and conversation history. We also find that although GPT-4 is usually more trustworthy than GPT-3.5 on standard benchmarks, GPT-4 is more vulnerable given jailbreaking system or user prompts, potentially because GPT-4 follows (misleading) instructions more precisely. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps. Our benchmark is publicly available at https://decodingtrust.github.io/ ; our dataset can be previewed at https://huggingface.co/datasets/AI-Secure/DecodingTrust ; a concise version of this work is at https://openreview.net/pdf?id=kaHpo8OZw2 .

翻译：生成式预训练Transformer（GPT）模型在其能力上展现出令人振奋的进步，吸引了从业者和公众的关注。然而，尽管关于GPT模型可信度的文献仍十分有限，从业者已提议将能力强大的GPT模型应用于医疗和金融等敏感领域——这些领域中的错误代价高昂。为此，本文提出一种针对大型语言模型的全面可信度评估，重点聚焦GPT-4和GPT-3.5，并从多个维度进行考量——包括毒性、刻板偏见、对抗鲁棒性、分布外鲁棒性、对抗演示鲁棒性、隐私、机器伦理以及公平性。基于我们的评估，我们发现了此前未公开的可信度威胁漏洞。例如，我们发现GPT模型容易被误导生成有毒和有偏见的内容，并泄露训练数据和对话历史中的隐私信息。我们还发现，尽管GPT-4在标准基准测试中通常比GPT-3.5更可信，但在面对越狱系统或用户提示时，GPT-4更容易受到攻击，这可能是因为GPT-4更精确地遵循（误导性）指令。我们的工作展示了GPT模型的全面可信度评估，并揭示了可信度方面的差距。我们的基准测试结果已公开于https://decodingtrust.github.io/；数据集可在https://huggingface.co/datasets/AI-Secure/DecodingTrust预览；本工作的精简版可在https://openreview.net/pdf?id=kaHpo8OZw2获取。