The rapid development in the field of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents to assist humans in their daily tasks. However, a significant gap remains in assessing whether LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the pressing need for methods to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval provides an implementation for the math problems}, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the robustness of quantifier's work.

翻译：大型语言模型（LLMs）领域的快速发展催生了大量应用，这些应用促进多智能体协作以协助人类完成日常任务。然而，在评估基于LLM的应用是否真正提升用户体验与任务执行效率方面仍存在显著空白。这凸显了验证基于LLM应用效用的迫切需求，特别是确保应用功能与终端用户需求的对齐。我们提出AgentEval——一种新颖的框架，旨在通过自动提出一套针对任意应用独特目标量身定制的评估标准，简化效用验证过程。该框架能够依据建议标准对应用效用进行量化评估。我们呈现了对量化器工作鲁棒性的全面分析。