With the rapid rise of AI coding agents, the fundamental premise of what it means to be a software engineer is in question. In this vision paper, we re-examine what it means for an AI agent to be considered a software engineer and then critically think about what makes such an agent trustworthy. \textit{Grounded} in established definitions of software engineering (SE) and informed by recent research on agentic AI systems, we conceptualise AI software engineers as participants in human-AI SE teams composed of human software engineers and AI models and tools, and we distinguish trustworthiness as a key property of these systems and actors rather than a subjective human attitude. Based on historical perspectives and emerging visions, we identify key dimensions that contribute to the trustworthiness of AI software engineers, spanning technical quality, transparency and accountability, epistemic humility, and societal and ethical alignment. We further discuss how trustworthiness can be evaluated and demonstrated, highlighting a fundamental trust measurement gap: not everything that matters for trust can be easily measured. Finally, we outline implications for the design, evaluation, and governance of AI SE systems, advocating for an ethics-by-design approach to enable appropriate trust in future human-AI SE teams.

翻译：随着AI编码代理的迅速兴起，关于软件工程师本质的基本前提正受到质疑。在这篇愿景论文中，我们重新审视了AI代理被视为软件工程师的含义，并批判性地思考了使此类代理值得信赖的关键要素。\textit{基于}软件工程（SE）的既定定义，并借鉴近期关于智能体AI系统的研究，我们将AI软件工程师概念化为由人类软件工程师与AI模型及工具组成的人机SE团队中的参与者，并将可信赖性界定为这些系统与行为体的关键属性，而非人类的主观态度。基于历史视角与新兴愿景，我们识别了构成AI软件工程师可信赖性的关键维度，涵盖技术质量、透明度与可问责性、认知谦逊以及社会与伦理对齐。我们进一步探讨了如何评估与证明可信赖性，同时指出了一个根本性的信任度量鸿沟：并非所有对信任至关重要的因素都能被轻易度量。最后，我们概述了这对AI SE系统的设计、评估与治理的启示，倡导通过"设计即伦理"的方法，为未来人机SE团队建立恰当的信任奠定基础。