Rapid advancements in artificial intelligence (AI) have sparked growing concerns among experts, policymakers, and world leaders regarding the potential for increasingly advanced AI systems to pose catastrophic risks. Although numerous risks have been detailed separately, there is a pressing need for a systematic discussion and illustration of the potential dangers to better inform efforts to mitigate them. This paper provides an overview of the main sources of catastrophic AI risks, which we organize into four categories: malicious use, in which individuals or groups intentionally use AIs to cause harm; AI race, in which competitive environments compel actors to deploy unsafe AIs or cede control to AIs; organizational risks, highlighting how human factors and complex systems can increase the chances of catastrophic accidents; and rogue AIs, describing the inherent difficulty in controlling agents far more intelligent than humans. For each category of risk, we describe specific hazards, present illustrative stories, envision ideal scenarios, and propose practical suggestions for mitigating these dangers. Our goal is to foster a comprehensive understanding of these risks and inspire collective and proactive efforts to ensure that AIs are developed and deployed in a safe manner. Ultimately, we hope this will allow us to realize the benefits of this powerful technology while minimizing the potential for catastrophic outcomes.

翻译：人工智能（AI）的快速发展引发了专家、政策制定者和世界领导人对日益先进的AI系统可能带来灾难性风险的日益担忧。尽管已有大量研究分别详述了多种风险，但当前迫切需要对这些潜在危害进行系统性论述和阐释，以便更有效地指导风险缓解工作。本文概述了灾难性AI风险的主要来源，并将其分为四类：恶意使用——个人或群体故意利用AI造成危害；AI竞赛——竞争环境迫使参与者部署不安全的AI或向AI让渡控制权；组织风险——强调人为因素与复杂系统如何增加灾难性事故发生的概率；以及失控AI——描述控制比人类智能得多的智能体所固有的困难。针对每类风险，我们描述了具体危害，呈现了示例性场景，构想了理想场景，并提出了缓解这些危险的实用建议。我们的目标是促进对这些风险的全面理解，并激发集体和积极主动的努力，以确保AI以安全的方式开发和部署。最终，我们希望这能使我们在最大限度降低潜在灾难性后果的同时，实现这一强大技术的益处。