In this 4-page manuscript we discuss the problem of long-term AI Safety from a Software Engineering (SE) research viewpoint. We briefly summarize long-term AI Safety, and the challenge of avoiding harms from AI as systems meet or exceed human capabilities, including software engineering capabilities (and approach AGI / "HLMI"). We perform a quantified literature review suggesting that AI Safety discussions are not common at SE venues. We make conjectures about how software might change with rising capabilities, and categorize "subproblems" which fit into traditional SE areas, proposing how work on similar problems might improve the future of AI and SE.

翻译：在这篇4页手稿中，我们从软件工程研究视角探讨了长期AI安全问题。我们简要总结了长期AI安全的内涵，以及当AI系统达到或超越人类能力（包括软件工程能力并接近通用人工智能/高级机器学习智能）时避免其造成危害的挑战。通过量化文献综述，我们发现AI安全议题在软件工程领域尚不常见。我们推测了随着AI能力提升软件可能发生的变化，并分类归纳了传统软件工程领域中的"子问题"，提出针对类似问题的研究如何能够改善未来AI与软件工程的发展。