By virtue of being prevalently written in natural language (NL), requirements are prone to various defects, e.g., inconsistency and incompleteness. As such, requirements are frequently subject to quality assurance processes. These processes, when carried out entirely manually, are tedious and may further overlook important quality issues due to time and budget pressures. In this paper, we propose QAssist -- a question-answering (QA) approach that provides automated assistance to stakeholders, including requirements engineers, during the analysis of NL requirements. Posing a question and getting an instant answer is beneficial in various quality-assurance scenarios, e.g., incompleteness detection. Answering requirements-related questions automatically is challenging since the scope of the search for answers can go beyond the given requirements specification. To that end, QAssist provides support for mining external domain-knowledge resources. Our work is one of the first initiatives to bring together QA and external domain knowledge for addressing requirements engineering challenges. We evaluate QAssist on a dataset covering three application domains and containing a total of 387 question-answer pairs. We experiment with state-of-the-art QA methods, based primarily on recent large-scale language models. In our empirical study, QAssist localizes the answer to a question to three passages within the requirements specification and within the external domain-knowledge resource with an average recall of 90.1% and 96.5%, respectively. QAssist extracts the actual answer to the posed question with an average accuracy of 84.2%. Keywords: Natural-language Requirements, Question Answering (QA), Language Models, Natural Language Processing (NLP), Natural Language Generation (NLG), BERT, T5.
翻译:摘要:由于需求通常以自然语言(NL)形式撰写,因此容易存在各种缺陷,例如不一致性和不完整性。为此,需求经常需要经过质量保证流程。这些流程若完全由人工执行,既繁琐又可能因时间和预算压力而忽略重要的质量问题。本文提出QAssist——一种问答(QA)方法,可在分析自然语言需求的过程中,为包括需求工程师在内的利益相关者提供自动化辅助。提出问题并即时获取答案有助于多种质量保证场景,例如不完整性检测。自动回答与需求相关的问题具有挑战性,因为答案的搜索范围可能超出给定的需求规格说明。为此,QAssist提供了挖掘外部领域知识资源的支持。本研究是将QA与外部领域知识相结合以应对需求工程挑战的首批尝试之一。我们在涵盖三个应用领域、共计387个问答对的数据集上评估了QAssist。我们基于近期大型语言模型,采用了最先进的QA方法进行实验。实证研究表明,QAssist将问题的答案定位至需求规格说明和外部领域知识资源中的三个段落,平均召回率分别达到90.1%和96.5%。QAssist提取所提出问题的实际答案的平均准确率为84.2%。关键词:自然语言需求,问答(QA),语言模型,自然语言处理(NLP),自然语言生成(NLG),BERT,T5。