The development of modern NLP applications often relies on various benchmark datasets containing plenty of manually labeled tests to evaluate performance. While constructing datasets often costs many resources, the performance on the held-out data may not properly reflect their capability in real-world application scenarios and thus cause tremendous misunderstanding and monetary loss. To alleviate this problem, in this paper, we propose an automated test generation method for detecting erroneous behaviors of various NLP applications. Our method is designed based on the sentence parsing process of classic linguistics, and thus it is capable of assembling basic grammatical elements and adjuncts into a grammatically correct test with proper oracle information. We implement this method into NLPLego, which is designed to fully exploit the potential of seed sentences to automate the test generation. NLPLego disassembles the seed sentence into the template and adjuncts and then generates new sentences by assembling context-appropriate adjuncts with the template in a specific order. Unlike the taskspecific methods, the tests generated by NLPLego have derivation relations and different degrees of variation, which makes constructing appropriate metamorphic relations easier. Thus, NLPLego is general, meaning it can meet the testing requirements of various NLP applications. To validate NLPLego, we experiment with three common NLP tasks, identifying failures in four state-of-art models. Given seed tests from SQuAD 2.0, SST, and QQP, NLPLego successfully detects 1,732, 5301, and 261,879 incorrect behaviors with around 95.7% precision in three tasks, respectively.

翻译：摘要：现代自然语言处理（NLP）应用的开发通常依赖包含大量人工标注测试样本的各类基准数据集来评估性能。然而，构建数据集往往耗费大量资源，且基于预留数据的评估结果可能无法准确反映模型在真实场景中的表现能力，进而导致严重误解与经济损失。为缓解该问题，本文提出一种用于检测各类NLP应用错误行为的自动化测试生成方法。该方法基于经典语言学中的句法分析过程设计，能够将基本语法成分与附加成分组装为语法正确且包含恰当预言信息（oracle information）的测试用例。我们将其实现为NLPLego系统——该系统旨在充分挖掘种子句的潜在能力以实现测试自动化生成。NLPLego将种子句分解为模板与附加成分，随后通过按特定顺序将与上下文适配的附加成分组装至模板，生成新测试用例。不同于面向特定任务的方法，NLPLego生成的测试具有派生关系及不同程度的变异特性，这使得构建合适的蜕变关系（metamorphic relations）更为简便。因此，NLPLego具有通用性，可满足各类NLP应用的测试需求。为验证NLPLego的有效性，我们针对三项常见NLP任务进行实验，检测了四种先进模型中的错误行为。基于来自SQuAD 2.0、SST和QQP的种子测试，NLPLego在这三项任务中分别成功检测出1,732、5,301和261,879个错误行为，精确率均约为95.7%。