Although it has been demonstrated that Natural Language Processing (NLP) algorithms are vulnerable to deliberate attacks, the question of whether such weaknesses can lead to software security threats is under-explored. To bridge this gap, we conducted vulnerability tests on Text-to-SQL systems that are commonly used to create natural language interfaces to databases. We showed that the Text-to-SQL modules within six commercial applications can be manipulated to produce malicious code, potentially leading to data breaches and Denial of Service attacks. This is the first demonstration that NLP models can be exploited as attack vectors in the wild. In addition, experiments using four open-source language models verified that straightforward backdoor attacks on Text-to-SQL systems achieve a 100% success rate without affecting their performance. The aim of this work is to draw the community's attention to potential software security issues associated with NLP algorithms and encourage exploration of methods to mitigate against them.

翻译：尽管已有研究证明自然语言处理（NLP）算法易受蓄意攻击，但这些漏洞是否可能引发软件安全威胁的问题尚待探索。为填补这一空白，我们对常用于构建数据库自然语言界面的文本到SQL系统进行了脆弱性测试。结果表明，六款商业应用中的文本到SQL模块可被操控生成恶意代码，可能导致数据泄露和拒绝服务攻击。这是首次证明NLP模型可在实际环境中被利用为攻击向量。此外，基于四个开源语言模型的实验证实，针对文本到SQL系统的简单后门攻击可实现100%的成功率且不影响其性能。本研究旨在引发学界对NLP算法相关潜在软件安全问题的关注，并鼓励探索相应的防御方法。