What is the prospect of developing artificial general intelligence (AGI)? I investigate this question by systematically comparing living and algorithmic systems, with a special focus on the notion of "agency." There are three fundamental differences to consider: (1) Living systems are autopoietic, that is, self-manufacturing, and therefore able to set their own intrinsic goals, while algorithms exist in a computational environment with target functions that are both provided by an external agent. (2) Living systems are embodied in the sense that there is no separation between their symbolic and physical aspects, while algorithms run on computational architectures that maximally isolate software from hardware. (3) Living systems experience a large world, in which most problems are ill-defined (and not all definable), while algorithms exist in a small world, in which all problems are well-defined. These three differences imply that living and algorithmic systems have very different capabilities and limitations. In particular, it is extremely unlikely that true AGI (beyond mere mimicry) can be developed in the current algorithmic framework of AI research. Consequently, discussions about the proper development and deployment of algorithmic tools should be shaped around the dangers and opportunities of current narrow AI, not the extremely unlikely prospect of the emergence of true agency in artificial systems.

翻译：发展通用人工智能（AGI）的前景如何？本文通过系统比较生命系统与算法系统，并聚焦于“能动性”概念来探讨这一问题。需考虑三个根本性差异：（1）生命系统是自创生的，即自我制造，因此能设定自身内在目标；而算法存在于计算环境中，其目标函数均由外部主体提供。（2）生命系统具有具身性，其符号层面与物理层面无法分离；而算法运行在最大化软硬件隔离的计算架构上。（3）生命系统经历的是大世界，其中多数问题定义不良（且并非所有问题均可定义）；而算法存在于小世界，其中所有问题均有良好定义。这三项差异表明，生命系统与算法系统在能力与局限性上存在本质区别。尤其值得指出的是，在现有AI研究的算法框架下，发展出真正的AGI（而非仅仅是拟态）的可能性极低。因此，关于算法工具合理开发与部署的讨论，应围绕当前狭义AI的风险与机遇展开，而非聚焦于人工系统涌现真正能动性这一极不现实的展望。