Recent advances in task-oriented dialogue (TOD) systems, driven by large language models (LLMs) with extensive API and tool integration, have enabled conversational agents to coordinate interleaved goals, maintain long-horizon context, and act proactively through asynchronous execution. These capabilities extend beyond traditional TOD systems, yet existing benchmarks lack systematic support for evaluating such agentic behaviors. To address this gap, we introduce ATOD, a benchmark and synthetic dialogue generation pipeline that produces richly annotated conversations requiring long-term reasoning. ATOD captures key characteristics of advanced TOD, including multi-goal coordination, dependency management, memory, adaptability, and proactivity. Building on ATOD, we propose ATOD-Eval, a holistic evaluation framework that translates these dimensions into fine-grained metrics and supports reproducible offline and online evaluation. We further present a strong agentic memory-based evaluator for benchmarking on ATOD. Experiments show that ATOD-Eval enables comprehensive assessment across task completion, agentic capability, and response quality, and that the proposed evaluator offers a better accuracy-efficiency tradeoff compared to existing memory- and LLM-based approaches under this evaluation setting.

翻译：近年来，借助大规模语言模型（LLMs）与广泛的API及工具集成，任务型对话系统取得了显著进展，使得对话智能体能够协调交织的目标、维持长程上下文，并通过异步执行主动采取行动。这些能力超越了传统任务型对话系统的范畴，然而现有基准缺乏对这类智能体行为的系统性评估支持。为填补这一空白，我们提出了ATOD——一个基准及合成对话生成流程，能够生成需要长期推理且带有丰富标注的对话。ATOD捕捉了先进任务型对话系统的关键特征，包括多目标协调、依赖管理、记忆、适应性与主动性。基于ATOD，我们进一步提出了ATOD-Eval，这是一个整体性评估框架，将上述维度转化为细粒度指标，并支持可复现的离线和在线评估。我们还提出了一种基于智能体记忆的强评估器，用于在ATOD上进行基准测试。实验表明，ATOD-Eval能够在任务完成度、智能体能力与响应质量方面实现全面评估，并且在此评估设置下，所提出的评估器相较于现有的基于记忆和基于LLM的方法，提供了更优的准确性与效率权衡。