Recent advances in task-oriented dialogue (TOD) systems, driven by large language models (LLMs) with extensive API and tool integration, have enabled conversational agents to coordinate interleaved goals, maintain long-horizon context, and act proactively through asynchronous execution. These capabilities extend beyond traditional TOD systems, yet existing benchmarks lack systematic support for evaluating such agentic behaviors. To address this gap, we introduce ATOD, a benchmark and synthetic dialogue generation pipeline that produces richly annotated conversations requiring long-term reasoning. ATOD captures key characteristics of advanced TOD, including multi-goal coordination, dependency management, memory, adaptability, and proactivity. Building on ATOD, we propose ATOD-Eval, a holistic evaluation framework that translates these dimensions into fine-grained metrics and supports reproducible offline and online evaluation. We further present a strong agentic memory-based evaluator for benchmarking on ATOD. Experiments show that ATOD-Eval enables comprehensive assessment across task completion, agentic capability, and response quality, and that the proposed evaluator offers a better accuracy-efficiency tradeoff compared to existing memory- and LLM-based approaches under this evaluation setting.

翻译：近年来，借助大规模语言模型（LLM）与丰富的API及工具集成，任务型对话（TOD）系统取得了显著进展，使得对话智能体能够协调交错的目标、维持长程上下文，并通过异步执行实现主动行为。这些能力已超越传统TOD系统的范畴，然而现有基准缺乏对这类智能体化行为的系统性评估支持。为填补这一空白，我们提出了ATOD——一个基准及合成对话生成流水线，能够生成需要长期推理的、带有丰富标注的对话。ATOD捕捉了先进TOD系统的关键特征，包括多目标协调、依赖关系管理、记忆能力、适应性与主动性。基于ATOD，我们进一步提出ATOD-Eval，一个将上述维度转化为细粒度指标、并支持可复现的离线和在线评估的整体性评估框架。我们还提出了一种基于智能体记忆的强评估器，用于在ATOD上进行基准测试。实验表明，ATOD-Eval能够在任务完成度、智能体能力与响应质量方面实现全面评估，且在此评估设置下，所提出的评估器相比现有的基于记忆和LLM的方法，提供了更优的准确性与效率权衡。