Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs. When such a mismatch occurs, the CoT no longer faithfully reflects the decision-critical factors driving the model's behavior, leading to the reduced CoT monitorability problem. However, a comprehensive and fully open-source benchmark for studying CoT monitorability remains lacking. To address this gap, we propose MonitorBench, a systematic benchmark for evaluating CoT monitorability in LLMs. MonitorBench provides: (1) a diverse set of 1,514 test instances with carefully designed decision-critical factors across 19 tasks spanning 7 categories to characterize when CoTs can be used to monitor the factors driving LLM behavior; and (2) two stress-test settings to quantify the extent to which CoT monitorability can be degraded. Extensive experiments across multiple popular LLMs with varying capabilities show that CoT monitorability is higher when producing the final target response requires structural reasoning through the decision-critical factor. Closed-source LLMs generally show lower monitorability, and there exists a negative relationship between monitorability and model capability. Moreover, both open- and closed-source LLMs can intentionally reduce monitorability under stress-tests, with monitorability dropping by up to 30% in some tasks that do not require structural reasoning over the decision-critical factors. Beyond these empirical insights, MonitorBench provides a basis for further research on evaluating future LLMs, studying advanced stress-test monitorability techniques, and developing new monitoring approaches.

翻译：大型语言模型（LLMs）生成的思维链（CoTs）并非总是与其最终输出具有因果对应关系。当此类不匹配发生时，思维链不再忠实反映驱动模型行为的决策关键因素，导致思维链可监控性下降问题。然而，目前仍缺乏系统全面且完全开源的思维链可监控性研究基准。为填补这一空白，我们提出MonitorBench——一个用于评估LLMs思维链可监控性的系统性基准。MonitorBench提供：（1）涵盖7大类19项任务的1514个多样化测试实例，其中精心设计的决策关键因素可表征思维链何时能被用于监控驱动LLM行为的因素；（2）两种压力测试设置，用于量化思维链可监控性的退化程度。在多个具有不同能力的流行LLM上开展的广泛实验表明：当最终目标响应的生成需要基于决策关键因素进行结构性推理时，思维链可监控性更高。闭源LLM普遍显示出较低的可监控性，且可监控性与模型能力之间存在负相关关系。此外，开源与闭源LLM均能在压力测试中主动降低可监控性，在部分无需对决策关键因素进行结构推理的任务中，可监控性下降幅度高达30%。除这些实证发现外，MonitorBench还为评估未来LLM、研究高级压力测试可监控性技术以及开发新型监控方法提供了研究基础。