The miscalibration of Large Reasoning Models (LRMs) undermines their reliability in high-stakes domains, necessitating methods to accurately estimate the confidence of their long-form, multi-step outputs. To address this gap, we introduce the Reasoning Model Confidence estimation Benchmark (RMCB), a public resource of 347,496 reasoning traces from six popular LRMs across different architectural families. The benchmark is constructed from a diverse suite of datasets spanning high-stakes domains, including clinical, financial, legal, and mathematical reasoning, alongside complex general reasoning benchmarks, with correctness annotations provided for all samples. Using RMCB, we conduct a large-scale empirical evaluation of over ten distinct representation-based methods, spanning sequential, graph-based, and text-based architectures. Our central finding is a persistent trade-off between discrimination (AUROC) and calibration (ECE): text-based encoders achieve the best AUROC (0.672), while structurally-aware models yield the best ECE (0.148), with no single method dominating both. Furthermore, we find that increased architectural complexity does not reliably outperform simpler sequential baselines, suggesting a performance ceiling for methods relying solely on chunk-level hidden states. This work provides the most comprehensive benchmark for this task to date, establishing rigorous baselines and demonstrating the limitations of current representation-based paradigms.

翻译：大型推理模型（LRMs）的校准失准问题削弱了其在高风险领域应用的可靠性，因此亟需开发能够准确评估其长文本、多步骤输出置信度的方法。为填补这一空白，我们提出了推理模型置信度估计基准（RMCB），这是一个包含来自六大主流LRMs（涵盖不同架构家族）共347,496条推理轨迹的公共资源库。该基准构建于跨越高风险领域的多样化数据集，涵盖临床诊断、金融分析、法律推理和数学演算等专业领域，同时包含复杂通用推理基准，所有样本均提供正确性标注。基于RMCB，我们对十余种基于表征的估计方法进行了大规模实证评估，涵盖序列模型、图结构模型和文本编码器等多种架构。核心研究发现：判别性能（AUROC）与校准性能（ECE）存在持续性权衡——文本编码器获得最佳AUROC（0.672），而结构感知模型实现最优ECE（0.148），没有任何单一方法能在两项指标上同时占优。此外，我们发现增加架构复杂度并不能稳定超越简单的序列基线方法，这表明仅依赖片段级隐藏状态的估计方法存在性能瓶颈。本研究构建了当前该任务最全面的基准体系，确立了严谨的基线标准，并揭示了现有基于表征的范式存在的局限性。