In this 4-page manuscript we discuss the problem of long-term AI Safety from a Software Engineering (SE) research viewpoint. We briefly summarize long-term AI Safety, and the challenge of avoiding harms from AI as systems meet or exceed human SE capabilities, including software engineering capabilities (and approach AGI / "HLMI"). We perform a quantified literature review suggesting that AI Safety discussions are not common at SE venues. We make conjectures about how software might change with rising capabilities, and categorize "subproblems" which fit into traditional SE areas, proposing how work on similar problems might improve the future of AI and SE.

翻译：在这篇4页的手稿中，我们从软件工程（SE）研究视角探讨长期AI安全问题。我们简要概述了长期AI安全，以及当AI系统达到或超越人类软件工程能力（包括软件工程能力，并接近AGI/"HLMI"）时避免其带来危害的挑战。我们通过量化文献综述表明，AI安全讨论在SE领域并不常见。我们推测了软件能力提升可能带来的变化，并对符合传统SE领域的"子问题"进行分类，提出针对类似问题的研究如何改善AI与SE的未来。