Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks do not adequately assess their ability to solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPSYNTH is constructed using a multi-stage data collection pipeline that requires annotators to collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as a crucial benchmark for guiding future research.

翻译：基于大语言模型（LLM）的智能体正日益广泛地应用于解决涉及工具使用的复杂任务，例如网页浏览、代码执行和数据分析。然而，当前的评估基准未能充分评估其解决现实世界任务的能力，这些任务需要综合来自多个来源的信息，并推断出超越简单事实检索的深层洞见。为此，我们提出了DEEPSYNTH，这是一个新颖的基准测试，旨在评估智能体在解决需要结合信息收集、综合与结构化推理以产生洞见的、耗时且贴近现实的问题上的表现。DEEPSYNTH包含跨7个领域和数据源收集的120项任务，覆盖67个国家。该基准通过一个多阶段数据收集流程构建，要求标注人员收集官方数据源、提出假设、进行人工分析并设计具有可验证答案的任务。在DEEPSYNTH上进行评估时，11个最先进的大语言模型与深度研究智能体在LLM-judge指标上取得的最高F1分数分别为8.97和17.5，凸显了该基准的难度。我们的分析表明，当前智能体在处理大规模信息空间时存在幻觉与推理困难，这突显了DEEPSYNTH作为指导未来研究的关键基准价值。