The dependence of Natural Language Processing (NLP) intelligent software on Large Language Models (LLMs) is increasingly prominent, underscoring the necessity for robustness testing. Current testing methods focus solely on the robustness of LLM-based software to prompts. Given the complexity and diversity of real-world inputs, studying the robustness of LLMbased software in handling comprehensive inputs (including prompts and examples) is crucial for a thorough understanding of its performance. To this end, this paper introduces RITFIS, a Robust Input Testing Framework for LLM-based Intelligent Software. To our knowledge, RITFIS is the first framework designed to assess the robustness of LLM-based intelligent software against natural language inputs. This framework, based on given threat models and prompts, primarily defines the testing process as a combinatorial optimization problem. Successful test cases are determined by a goal function, creating a transformation space for the original examples through perturbation means, and employing a series of search methods to filter cases that meet both the testing objectives and language constraints. RITFIS, with its modular design, offers a comprehensive method for evaluating the robustness of LLMbased intelligent software. RITFIS adapts 17 automated testing methods, originally designed for Deep Neural Network (DNN)-based intelligent software, to the LLM-based software testing scenario. It demonstrates the effectiveness of RITFIS in evaluating LLM-based intelligent software through empirical validation. However, existing methods generally have limitations, especially when dealing with lengthy texts and structurally complex threat models. Therefore, we conducted a comprehensive analysis based on five metrics and provided insightful testing method optimization strategies, benefiting both researchers and everyday users.

翻译：自然语言处理（NLP）智能软件对大型语言模型（LLM）的依赖日益显著，因而亟需开展鲁棒性测试。当前测试方法仅关注基于LLM的软件对提示的鲁棒性。鉴于真实世界输入的复杂性与多样性，研究基于LLM的软件在处理综合输入（包括提示和示例）时的鲁棒性，对于全面理解其性能至关重要。为此，本文提出RITFIS——面向基于LLM的智能软件的鲁棒输入测试框架。据我们所知，RITFIS是首个旨在评估基于LLM的智能软件对自然语言输入鲁棒性的框架。该框架基于给定威胁模型和提示，将测试过程定义为一个组合优化问题。通过目标函数确定成功测试用例，利用扰动手段创建原始示例的变换空间，并采用一系列搜索方法筛选既满足测试目标又符合语言约束的用例。RITFIS采用模块化设计，为评估基于LLM的智能软件鲁棒性提供了综合性方法。它将原本面向深度神经网络（DNN）智能软件的17种自动化测试方法适配至基于LLM的软件测试场景，并通过实证验证证明了RITFIS在评估基于LLM的智能软件中的有效性。然而，现有方法普遍存在局限性，尤其在处理长文本和结构复杂的威胁模型时。因此，我们基于五项指标进行了全面分析，并提出了富有洞见的测试方法优化策略，使研究人员和日常用户均能从中受益。