The advancement of Artificial Intelligence (AI) and Machine Learning (ML) has profound implications for both the utility and security of our digital interactions. This paper investigates the transformative role of Generative AI in Social Engineering (SE) attacks. We conduct a systematic review of social engineering and AI capabilities and use a theory of social engineering to identify three pillars where Generative AI amplifies the impact of SE attacks: Realistic Content Creation, Advanced Targeting and Personalization, and Automated Attack Infrastructure. We integrate these elements into a conceptual model designed to investigate the complex nature of AI-driven SE attacks - the Generative AI Social Engineering Framework. We further explore human implications and potential countermeasures to mitigate these risks. Our study aims to foster a deeper understanding of the risks, human implications, and countermeasures associated with this emerging paradigm, thereby contributing to a more secure and trustworthy human-computer interaction.

翻译：人工智能（AI）与机器学习（ML）的进步对我们的数字交互在实用性和安全性方面产生了深远影响。本文探讨了生成式人工智能在社交工程（SE）攻击中的变革性作用。我们系统梳理了社交工程与AI能力，并运用社交工程理论，识别出生成式AI增强SE攻击影响的三大支柱：逼真内容创建、精准目标定位与个性化定制，以及自动化攻击基础设施。我们将这些要素整合为一个概念模型——生成式人工智能社交工程框架，旨在探究AI驱动SE攻击的复杂性。我们进一步探讨了人类层面影响及潜在应对措施以减轻这些风险。本研究旨在促进对新兴范式相关风险、人类影响及应对措施的深入理解，从而为人机交互的安全可信发展做出贡献。