Detecting LLM reasoning failures at inference time without ground-truth labels has motivated a wide range of confidence baselines, including self-consistency, semantic entropy, and P(True), built on within-question sampling and self-evaluation. Operad theory, the formalism for systems built by iterated substitution, suggests a complementary diagnostic: a model's direct answer to a compositional query should agree with the answer it produces by composing a stated decomposition of the same query. We instantiate this idea as operadic consistency (OC), a per-question signal. Across twelve instruction-tuned LLMs (4B to 671B parameters, open-weights and closed-source) on four multi-hop QA datasets, OC is strongly correlated with accuracy on every dataset (Pearson $r \in [0.86, 0.94]$, all $p \leq 0.0004$), and is the only signal we evaluate with $r \geq 0.85$ uniformly across all four datasets. Chain-of-thought self-consistency (CoT-SC; Wang et al., 2023) matches OC on HotpotQA and DROP ($r = 0.93, 0.87$) but drops to $r \approx 0.45$ on MuSiQue and StrategyQA. At the per-question level, OC contributes information beyond CoT-SC and semantic entropy on every dataset (cluster-robust $p \leq 10^{-16}$ for the OC coefficient), and the conclusion is robust to additionally controlling for constructed decomposition-aware baselines ($p \leq 10^{-13}$). The same signal yields selective-prediction improvements (accuracy at fixed coverage) over a tuned CoT-SC baseline at the equal-cost $K = 3$ budget (AUARC lifts of +0.086 to +0.096 and AUROC lifts of +0.092 to +0.164; 95% CIs exclude zero on every cell). On five frontier thinking models, where the decomposition is extracted from the model's own chain of thought, the same equal-cost comparison gives positive selective-prediction point-estimate lift on all 16 (dataset, budget, metric) cells tested, with 95% CIs excluding zero on 12 of the 16.

翻译：在推理时无需真实标签即可检测大型语言模型（LLM）推理失败的需求，催生了多种置信度基线方法，包括基于问题内采样与自我评估的自我一致性、语义熵以及P(True)。操作理论（Operad theory）——一种用于描述通过迭代替换构建的系统的形式化框架——提供了一种互补的诊断方法：模型对组合查询的直接应答应与通过组合同一查询的分解式陈述所产生的应答保持一致。我们将这一思想具体化为运算一致性（Operadic Consistency, OC），即每个问题层面的信号。在四个多跳问答数据集上对十二个经过指令微调的LLM（参数规模从4B到671B，包括开源权重与闭源模型）进行测试后发现，OC与所有数据集的准确率均呈现强相关性（Pearson相关系数r ∈ [0.86, 0.94]，所有p值≤0.0004），且是唯一在所有四个数据集上均一致达到r ≥ 0.85的信号。思维链自我一致性（CoT-SC; Wang et al., 2023）在HotpotQA和DROP数据集上与OC表现相当（r = 0.93, 0.87），但在MuSiQue和StrategyQA上骤降至r ≈ 0.45。在每个问题层面，OC在所有数据集上均提供了超越CoT-SC和语义熵的额外信息（OC系数的聚类稳健p值≤ 10^{-16}），且该结论在额外控制基于分解的基线方法后依然稳健（p ≤ 10^{-13}）。同一信号在成本相等的K=3预算下，相较于经过调优的CoT-SC基线方法，能够带来选择性预测性能的提升（固定覆盖率下的准确率提升：AUARC增益为+0.086至+0.096，AUROC增益为+0.092至+0.164；每组95%置信区间均不包含零）。在五个前沿推理模型（其分解过程从模型自身的思维链中提取）上，相同成本比较结果显示：在所有16个（数据集、预算、指标）测试组合中，选择性预测的点估计均呈现正向增益，其中12个组合的95%置信区间不包含零。