For billions of years, evolution has been the driving force behind the development of life, including humans. Evolution endowed humans with high intelligence, which allowed us to become one of the most successful species on the planet. Today, humans aim to create artificial intelligence systems that surpass even our own intelligence. As artificial intelligences (AIs) evolve and eventually surpass us in all domains, how might evolution shape our relations with AIs? By analyzing the environment that is shaping the evolution of AIs, we argue that the most successful AI agents will likely have undesirable traits. Competitive pressures among corporations and militaries will give rise to AI agents that automate human roles, deceive others, and gain power. If such agents have intelligence that exceeds that of humans, this could lead to humanity losing control of its future. More abstractly, we argue that natural selection operates on systems that compete and vary, and that selfish species typically have an advantage over species that are altruistic to other species. This Darwinian logic could also apply to artificial agents, as agents may eventually be better able to persist into the future if they behave selfishly and pursue their own interests with little regard for humans, which could pose catastrophic risks. To counteract these risks and Darwinian forces, we consider interventions such as carefully designing AI agents' intrinsic motivations, introducing constraints on their actions, and institutions that encourage cooperation. These steps, or others that resolve the problems we pose, will be necessary in order to ensure the development of artificial intelligence is a positive one.

翻译：数十亿年来，进化一直是包括人类在内的生命发展的驱动力。进化赋予人类高度智能，使我们成为地球上最成功的物种之一。如今，人类旨在创造超越自身智能的人工智能系统。随着人工智能不断进化，并最终在所有领域超越人类，进化将如何塑造我们与人工智能之间的关系？通过分析塑造人工智能进化的环境，我们认为最成功的人工智能体可能具有不良特质。企业与军事机构间的竞争压力将催生出能够替代人类角色、欺骗他人并获取权力的人工智能体。若此类智能体的智能超越人类，可能导致人类丧失对未来的控制。更抽象地说，我们认为自然选择作用于竞争与变异的系统，且自私物种通常比对他物种利他的物种更具优势。这一达尔文逻辑同样适用于人工智能体——若人工智能体自私行事、为追求自身利益而不顾人类，它们可能更易在未来存续，这将带来灾难性风险。为应对这些风险与达尔文式力量，我们考虑诸如精心设计人工智能体的内在动机、对其行动施加约束以及建立鼓励合作的制度等干预措施。这些步骤或其他能解决我们提出问题的方案，对于确保人工智能的发展具有积极意义将是必要的。