Verbatim feedback constitutes a valuable repository of user experiences, opinions, and requirements essential for software development. Effectively and efficiently extracting valuable insights from such data poses a challenging task. This paper introduces Allhands , an innovative analytic framework designed for large-scale feedback analysis through a natural language interface, leveraging large language models (LLMs). Allhands adheres to a conventional feedback analytic workflow, initially conducting classification and topic modeling on the feedback to convert them into a structurally augmented format, incorporating LLMs to enhance accuracy, robustness, generalization, and user-friendliness. Subsequently, an LLM agent is employed to interpret users' diverse questions in natural language on feedback, translating them into Python code for execution, and delivering comprehensive multi-modal responses, including text, code, tables, and images. We evaluate Allhands across three diverse feedback datasets. The experiments demonstrate that Allhands achieves superior efficacy at all stages of analysis, including classification and topic modeling, eventually providing users with an ``ask me anything'' experience with comprehensive, correct and human-readable response. To the best of our knowledge, Allhands stands as the first comprehensive feedback analysis framework that supports diverse and customized requirements for insight extraction through a natural language interface.

翻译：逐字反馈构成了软件开发中用户经验、意见和需求的宝贵资源。如何有效且高效地从这类数据中提取有价值信息仍是一项挑战。本文介绍Allhands，一个创新性分析框架，通过自然语言界面借助大型语言模型（LLMs）实现大规模反馈分析。Allhands遵循传统反馈分析流程，首先对反馈进行分类和主题建模，将其转换为结构增强格式，并结合LLMs提升准确性、鲁棒性、泛化能力及用户友好性。随后，采用LLM智能体解析用户针对反馈提出的多样化自然语言问题，将其转化为可执行的Python代码，并返回包含文本、代码、表格及图像的综合多模态响应。我们在三个不同反馈数据集上评估Allhands。实验表明，Allhands在分析各阶段（包括分类与主题建模）均展现出优越效能，最终为用户提供“有问必答”体验，输出全面、正确且可读的响应。据我们所知，Allhands是首个通过自然语言界面支持多样化定制需求以提取洞察的综合性反馈分析框架。