Large Language Models (LLMs) achieve strong performance on many reasoning benchmarks, yet these evaluations typically focus on isolated tasks that differ from real-world usage in task-oriented dialogue (TOD). In this setting, LLMs must perform reasoning inherently while generating text and adhering to instructions on role, format, and style. This mismatch raises concerns about whether benchmark performance accurately reflects models' reasoning robustness in TOD setting. We investigate how framing reasoning tasks within TOD affects LLM performance by introducing BOULDER, a new dynamic benchmark covering eight travel-related tasks that require arithmetic, spatial, and temporal reasoning with both commonsense and formal aspects. Each problem is presented in both isolated and dialogue-based variants, enabling controlled comparison while mitigating data contamination. Experiments on eight LLMs reveal a substantial and consistent performance gap between isolated and dialogue settings. Through ablations and qualitative analysis, we show that this gap is largely driven by the multi-turn nature of dialogue, with additional effects from role conditioning and tool-use requirements. Our results highlight the need to evaluate LLM reasoning in realistic interactive scenarios.

翻译：大语言模型（LLMs）在许多推理基准测试中展现出强劲性能，但这些评估通常聚焦于与任务导向型对话（TOD）真实使用场景不同的孤立任务。在此场景中，LLMs必须在生成文本、遵循角色、格式和风格指令的同时进行内隐推理。这种不匹配引发了关于基准测试性能能否准确反映模型在TOD场景中推理鲁棒性的担忧。我们通过引入BOULDER——一个涵盖算术、空间与时间推理（兼具常识性与形式化特征）的八项旅行相关任务的新型动态基准，探究了在TOD框架内设置推理任务对LLM性能的影响。每个问题均提供孤立与对话两种变体，可在缓解数据污染的同时实现可控对比。对八种LLMs的实验揭示出孤立设置与对话设置之间存在显著且一致的性能差距。通过消融实验与定性分析，我们表明该差距主要由对话的多轮特性驱动，同时角色条件设定与工具使用需求产生附加影响。研究结果凸显了在真实交互场景中评估LLM推理能力的必要性。