Large Language Model (LLM)-based applications are increasingly deployed across various domains, including customer service, education, and mobility. However, these systems are prone to inaccurate, fictitious, or harmful responses, and their vast, high-dimensional input space makes systematic testing particularly challenging. To address this, we present STELLAR, an automated search-based testing framework for LLM-based applications that systematically uncovers text inputs leading to inappropriate system responses. Our framework models test generation as an optimization problem and discretizes the input space into stylistic, content-related, and perturbation features. Unlike prior work that focuses on prompt optimization or coverage heuristics, our work employs evolutionary optimization to dynamically explore feature combinations that are more likely to expose failures. We evaluate STELLAR on three LLM-based conversational question-answering systems. The first focuses on safety, benchmarking both public and proprietary LLMs against malicious or unsafe prompts. The second and third target navigation, using an open-source and an industrial retrieval-augmented system for in-vehicle venue recommendations. Overall, STELLAR exposes up to 4.3 times (average 2.5 times) more failures than the existing baseline approaches.

翻译：基于大型语言模型（LLM）的应用正日益广泛地部署于客户服务、教育和移动出行等多个领域。然而，这些系统容易产生不准确、虚构或有害的回应，且其庞大、高维的输入空间使得系统性测试尤为困难。为此，我们提出了STELLAR——一个针对基于LLM的应用的自动化搜索测试框架，该系统性地发现导致不当系统响应的文本输入。本框架将测试生成建模为一个优化问题，并将输入空间离散化为风格、内容相关和扰动特征。与先前专注于提示优化或覆盖启发式方法的研究不同，我们的工作采用进化优化来动态探索更可能暴露故障的特征组合。我们在三个基于LLM的对话问答系统上评估了STELLAR。第一个系统聚焦于安全性，针对恶意或不安全的提示对公共和专有LLM进行基准测试。第二和第三个系统以导航为目标，分别使用开源和工业级检索增强系统进行车载场所推荐。总体而言，STELLAR暴露的故障数量最高可达现有基线方法的4.3倍（平均2.5倍）。