A systematic, reliable, and low-cost evaluation of Conversational Information Access (CIA) systems remains an open challenge. Existing reference-based evaluation methods are proven insufficient for evaluating the dynamic nature of information access conversations, while existing LLM-based reference-free methods suffer from evaluation bias and limited generalizability. This work proposes FACE: a Fine-grained, Aspect-based Conversation Evaluation method that provides evaluation scores for diverse turn and dialogue-level aspects of conversations. FACE leverages beam search and bandit optimization to select optimized LLM instructions per evaluation aspect. It assigns scores to atomic information units (particles) using the selected instructions and then aggregates them into a single score. We show that FACE achieves a strong correlation with human judgments, achieving system correlation of 0.9, outperforming state-of-the-art conversation evaluation methods by a large margin. We further demonstrate its optimized instructions are transferable across various LLMs and datasets. Additionally, unlike existing LLM-based methods that provide single uninterpretable scores, FACE provides insights into the system performance and enables identifying and locating problems within conversations.

翻译：对对话信息访问（CIA）系统进行系统性、可靠且低成本的评估仍是一项开放挑战。现有基于参考的评估方法被证明不足以评估信息访问对话的动态特性，而当前基于LLM的无参考方法则存在评估偏差和泛化能力有限的问题。本文提出FACE：一种细粒度、基于方面的对话评估方法，可为对话中多种轮次级和对话级方面提供评估分数。FACE利用束搜索（beam search）与置信域优化（bandit optimization）为每个评估方面选择优化的LLM指令。它使用选定指令对原子信息单元（particles）进行评分，并将这些分数聚合为单一分数。我们证明FACE与人类判断具有强相关性，在系统层面达到0.9的相关性，大幅超越现有最先进的对话评估方法。我们进一步证明其优化的指令可跨多种LLM和数据集迁移。此外，与仅提供单一不可解释分数的现有基于LLM的方法不同，FACE能提供对系统性能的洞见，并支持识别与定位对话中的问题。