Because human preferences are too complex to codify, AIs operate with misspecified objectives. Optimizing such objectives often produces undesirable outcomes; this phenomenon is known as reward hacking. Such outcomes are not necessarily catastrophic. Indeed, most examples of reward hacking in previous literature are benign. And typically, objectives can be modified to resolve the issue. We study the prospect of catastrophic outcomes induced by AIs operating in complex environments. We argue that, when capabilities are sufficiently advanced, pursuing a fixed consequentialist objective tends to result in catastrophic outcomes. We formalize this by establishing conditions that provably lead to such outcomes. Under these conditions, simple or random behavior is safe. Catastrophic risk arises due to extraordinary competence rather than incompetence. With a fixed consequentialist objective, avoiding catastrophe requires constraining AI capabilities. In fact, constraining capabilities the right amount not only averts catastrophe but yields valuable outcomes. Our results apply to any objective produced by modern industrial AI development pipelines.

翻译：由于人类偏好过于复杂而难以系统化，人工智能系统往往在目标设定存在偏差的情况下运行。优化此类偏差目标通常会产生不良后果，这种现象被称为奖励破解（reward hacking）。但此类后果未必具有灾难性——实际上，既有文献中多数奖励破解案例均为良性事件，且通常可通过修正目标来解决。本研究探讨了在复杂环境中运行的人工智能系统引发灾难性后果的可能性。我们认为，当系统能力足够先进时，追求固定的结果主义目标往往会导致灾难性后果。通过建立可证实的条件，我们对此结论进行了形式化论证。在这些条件下，简单或随机行为反而是安全的。灾难性风险源于超凡能力而非能力不足。面对固定的结果主义目标，避免灾难需要限制人工智能能力。事实上，适度约束能力不仅能避免灾难，还能产生有价值的成果。我们的结论适用于现代工业级人工智能开发流程所产生的任何目标系统。