The performance of large language models (LLMs) on verifiable tasks is usually measured by pass@k, the probability of answering a question correctly at least once in k trials. At a fixed budget, a more suitable metric is coverage@cost, the average number of unique questions answered as a function of the total number of attempts. We connect the two metrics and show that the empirically-observed power-law behavior in pass@k leads to a sublinear growth of the coverage@cost (diminishing returns). To solve this problem, we propose Reset-and-Discard (ReD), a query method of LLMs that increases coverage@cost for a given budget, regardless of the pass@k form. Moreover, given a pass@k, we can quantitatively predict the savings in the total number of attempts using ReD. If pass@k is not available for the model, ReD can infer its power-law exponent. Experiments on three LLMs across coding (HumanEval), math (GSM8K), and reasoning (MMLU-Pro) benchmarks demonstrate that ReD substantially reduces the required attempts, tokens, and USD cost to reach a desired coverage, while also offering an efficient way to measure inference power-laws. ReD's advantage is maintained for imperfect verifiers and outperforms the tested allocation baselines.

翻译：大型语言模型（LLMs）在可验证任务上的性能通常用pass@k来衡量，即在k次尝试中至少正确回答一次的概率。在固定预算下，更合适的指标是coverage@cost，即给定总尝试次数下被解答的不同问题的平均数量。我们关联了这两个指标，并表明pass@k中经验观测到的幂律行为会导致coverage@cost呈次线性增长（收益递减）。为解决此问题，我们提出了重置与丢弃（ReD）方法，这是一种LLM查询策略，能在给定预算下提高coverage@cost，且不受pass@k形式的影响。此外，给定pass@k，我们可以定量预测使用ReD节省的总尝试次数。若模型不具备pass@k数据，ReD可推断其幂律指数。在三个LLM的编码（HumanEval）、数学（GSM8K）和推理（MMLU-Pro）基准测试上的实验表明，ReD显著减少了达到目标覆盖率所需的尝试次数、令牌数及美元成本，同时提供了一种高效测量推理幂律的方法。ReD的优势在非完美验证器下仍能保持，并优于测试的分配基线方法。