Benchmarking AI systems in multi-turn interactive scenarios is essential for understanding their practical capabilities in real-world applications. However, existing evaluation protocols are highly heterogeneous, differing significantly in dataset formats, model interfaces, and evaluation pipelines, which severely impedes systematic comparison. In this work, we present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems. The core contribution of UDE lies in its holistic unification: it standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a consistent scoring interface. It also supports efficient large-scale evaluation through parallel generation and scoring, as well as checkpoint-based caching to eliminate redundant computation. Validated across diverse multi-turn benchmarks, UDE not only guarantees high reproducibility through standardized workflows and transparent logging, but also significantly improves evaluation efficiency and extensibility. We make the complete toolkit and evaluation scripts publicly available to foster a standardized benchmarking ecosystem and accelerate future breakthroughs in interactive AI.

翻译：在多轮交互场景中评估AI系统对于理解其在实际应用中的实用能力至关重要。然而，现有评估协议高度异构，在数据集格式、模型接口和评估流水线上存在显著差异，严重阻碍了系统性对比。本文提出UniDial-EvalKit（UDE），一个用于评估交互式AI系统的统一评估工具包。UDE的核心贡献在于其全方位的统一性：它将异构数据格式标准化为通用架构，通过模块化架构简化复杂的评估流水线，并在统一评分接口下对齐指标计算。该工具包还通过并行生成与评分支持高效的大规模评估，并采用基于检查点的缓存机制消除冗余计算。经多个多轮基准验证，UDE不仅通过标准化工作流与透明日志记录确保了高可复现性，还显著提升了评估效率与可扩展性。我们已公开提供完整工具包与评估脚本，旨在促进标准化的基准评估生态，加速交互式AI领域的未来突破。