Recent progress in artificial intelligence (AI) has drawn attention to the technology's transformative potential, including what some see as its prospects for causing large-scale harm. We review two influential arguments purporting to show how AI could pose catastrophic risks. The first argument -- the Problem of Power-Seeking -- claims that, under certain assumptions, advanced AI systems are likely to engage in dangerous power-seeking behavior in pursuit of their goals. We review reasons for thinking that AI systems might seek power, that they might obtain it, that this could lead to catastrophe, and that we might build and deploy such systems anyway. The second argument claims that the development of human-level AI will unlock rapid further progress, culminating in AI systems far more capable than any human -- this is the Singularity Hypothesis. Power-seeking behavior on the part of such systems might be particularly dangerous. We discuss a variety of objections to both arguments and conclude by assessing the state of the debate.

翻译：近年来人工智能（AI）的进展引发了人们对其变革潜力的关注，包括部分人认为其可能造成大规模危害的担忧。本文审视了两个具有影响力的论证，旨在说明AI如何可能引发灾难性风险。第一个论证——"权力追逐问题"——声称在特定假设下，先进AI系统倾向于在其目标追求过程中实施危险的权力追逐行为。我们系统梳理了AI系统可能追求权力、可能获取权力、这或将导致灾难性后果，以及我们仍可能构建并部署此类系统的多重依据。第二个论证则主张，通用人工智能的发展将解锁快速进步，最终催生远超人类能力的AI系统——此即"奇点假说"。这类系统的权力追逐行为可能尤为危险。我们讨论了针对这两种论证的多项质疑，最后通过评估当前辩论现状作出总结。