The discourse on risks from advanced AI systems ("AIs") typically focuses on misuse, accidents and loss of control, but the question of AIs' moral status could have negative impacts which are of comparable significance and could be realised within similar timeframes. Our paper evaluates these impacts by investigating (1) the factual question of whether future advanced AI systems will be conscious, together with (2) the epistemic question of whether future human society will broadly believe advanced AI systems to be conscious. Assuming binary responses to (1) and (2) gives rise to four possibilities: in the true positive scenario, society predominantly correctly believes that AIs are conscious; in the false positive scenario, that belief is incorrect; in the true negative scenario, society correctly believes that AIs are not conscious; and lastly, in the false negative scenario, society incorrectly believes that AIs are not conscious. The paper offers vivid vignettes of the different futures to ground the two-dimensional framework. Critically, we identify four major risks: AI suffering, human disempowerment, geopolitical instability, and human depravity. We evaluate each risk across the different scenarios and provide an overall qualitative risk assessment for each scenario. Our analysis suggests that the worst possibility is the wrong belief that AI is non-conscious, followed by the wrong belief that AI is conscious. The paper concludes with the main recommendations to avoid research aimed at intentionally creating conscious AI and instead focus efforts on reducing our current uncertainties on both the factual and epistemic questions on AI consciousness.

翻译：关于先进人工智能系统（"AIs"）风险的讨论通常集中于滥用、事故和失控问题，但人工智能的道德地位问题可能产生同等重要且可能在相似时间范围内显现的负面影响。本文通过研究以下两个问题评估这些影响：（1）未来先进人工智能系统是否具有意识的事实性问题，以及（2）未来人类社会是否广泛相信先进人工智能系统具有意识的认识论问题。对（1）和（2）采用二元化响应会产生四种可能性：在真阳性情境中，社会主流正确相信AI具有意识；在假阳性情境中，该信念是错误的；在真阴性情境中，社会正确相信AI不具有意识；最后，在假阴性情境中，社会错误地相信AI不具有意识。本文通过生动的未来情景描绘来夯实这个二维分析框架。关键的是，我们识别出四大风险：AI遭受痛苦、人类权力丧失、地缘政治不稳定以及人类道德堕落。我们评估了不同情境下各项风险的表现，并为每种情境提供了总体定性风险评估。分析表明，最糟糕的可能性是错误相信AI无意识，其次是错误相信AI有意识。论文最后提出核心建议：应避免旨在刻意创造有意识AI的研究，转而着力减少当前在AI意识问题上事实性与认知性两方面的不确定性。