Issues faced when using software are reported in the form of bug reports. However, many bug reports are invalid, meaning they do not require code changes, and are resolved with a no-code fix. Manually determining the root cause of the invalid bug reports and providing actionable resolutions by the customer support causes a serious waste of resources. Our goal is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification, and perform experiments to test the accuracy of various approaches on invalid subclassification and no-code fix generation. We study how different configurations perform on a gold-standard benchmark we have created. Using a manually curated benchmark for higher quality analysis, we experimented with vanilla LLMs, Retrieval Augmented Generation, and agentic web search to identify invalid subclasses and generate no-code fixes. We evaluated the results against manually labeled ground truth data that includes the invalid subclass and no-code fixes from the original bug reports. We measured subclass detection performance with weighted F1-Score, and assessed no-code fix suggestions using BERTScore and Judge LLM success rates. For subclassification, retrieval augmented generation achieves the highest overall performance with 0.66 weighted F1, slightly outperforming vanilla LLMs at 0.65 and agentic web search at 0.64. At the subclass level, performance peaks at 0.85 F1 for Non-reproducibility and 0.79 for Feature Request and Question, while Wrong Version remains the most challenging with scores between 0.00 and 0.29. For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68.9%, compared to 64.4% for RAG applications and 64.9% for vanilla LLMs, with subclass-level peaks of 87.4% for Working as Designed and 72.2% for Question.
翻译:软件使用过程中遇到的问题通常以Bug报告形式提交。然而,许多Bug报告是无效的,即它们不涉及代码变更,仅需通过无代码修复方式解决。由客服人员手动确定无效Bug报告的根因并提供可操作的解决方案会造成严重的资源浪费。我们的目标是引入一套面向根因的无效Bug报告子类划分标准化分类体系,并通过实验测试不同方法在无效子类划分和无代码修复生成任务中的准确性。我们研究了不同配置在我们创建的金标准基准上的性能表现。为进行更高质量的分析,我们基于人工整理的标准数据集,实验了纯净大语言模型、检索增强生成和智能体网络搜索三种方法,以识别无效子类并生成无代码修复方案。我们将结果与包含原始Bug报告中无效子类及无代码修复方案的人工标注真值数据进行对比。采用加权F1分数衡量子类检测性能,使用BERTScore和评判大语言模型成功率评估无代码修复建议。在子类划分任务中,检索增强生成整体性能最优,加权F1达0.66,略高于纯净大语言模型(0.65)和智能体网络搜索(0.64)。在子类层面,不可复现性问题以0.85的F1得分表现最佳,功能请求与咨询类达0.79,而版本错误仍最具挑战性,得分区间为0.00至0.29。在无代码修复生成任务中,智能体网络搜索的评判大语言模型整体成功率最高(68.9%),对比检索增强生成应用(64.4%)和纯净大语言模型(64.9%),其中按预期工作类的子类级峰值达87.4%,咨询类达72.2%。