Large-Language Models (LLMs) have shifted the paradigm of natural language data processing. However, their black-boxed and probabilistic characteristics can lead to potential risks in the quality of outputs in diverse LLM applications. Recent studies have tested Quality Attributes (QAs), such as robustness or fairness, of LLMs by generating adversarial input texts. However, existing studies have limited their coverage of QAs and tasks in LLMs and are difficult to extend. Additionally, these studies have only used one evaluation metric, Attack Success Rate (ASR), to assess the effectiveness of their approaches. We propose a MEtamorphic Testing for Analyzing LLMs (METAL) framework to address these issues by applying Metamorphic Testing (MT) techniques. This approach facilitates the systematic testing of LLM qualities by defining Metamorphic Relations (MRs), which serve as modularized evaluation metrics. The METAL framework can automatically generate hundreds of MRs from templates that cover various QAs and tasks. In addition, we introduced novel metrics that integrate the ASR method into the semantic qualities of text to assess the effectiveness of MRs accurately. Through the experiments conducted with three prominent LLMs, we have confirmed that the METAL framework effectively evaluates essential QAs on primary LLM tasks and reveals the quality risks in LLMs. Moreover, the newly proposed metrics can guide the optimal MRs for testing each task and suggest the most effective method for generating MRs.

翻译：大语言模型（LLMs）已彻底改变了自然语言数据处理范式，但其黑盒与概率特性可能导致各类LLM应用在输出质量上存在潜在风险。近期研究通过生成对抗性输入文本，对LLM的鲁棒性、公平性等质量属性（QAs）进行了测试。然而，现有研究在覆盖的LLM质量属性与任务类型方面存在局限，且难以扩展。此外，这些研究仅使用单一评估指标——攻击成功率（ASR）来衡量方法有效性。我们提出面向LLM的蜕变测试分析框架METAL，通过应用蜕变测试（MT）技术解决上述问题。该方法通过定义模块化评估指标——蜕变关系（MRs），实现了对LLM质量的系统性测试。METAL框架可从模板自动生成覆盖多种质量属性与任务类型的数百个MRs。同时，我们引入新型评估指标，将ASR方法与文本语义质量相结合，以精确评估MRs的有效性。通过在三个主流LLM上开展的实验，我们证实METAL框架能有效评估主要LLM任务中的关键质量属性，并揭示LLM存在的质量风险。此外，新提出的指标可指导为每项任务选择最优MRs，并推荐最有效的MRs生成方法。