Before an LLM agent can use a tool, a retrieval system must decide which candidate tools to show to the agent. How long should that shortlist be? Show too many tools and the model struggles to choose. Show too few and the correct tool may not appear. Most systems apply a fixed shortlist size to every query, but no standard metric exists to evaluate whether that size was appropriate. We treat the number of tools shown to an LLM agent as the object of evaluation and we apply Bits-over-Random (BoR), a chance-corrected metric that asks whether success at a given depth is better than what random selection would achieve at that same depth. We evaluate BoR across three tool-selection benchmarks, multiple scorers, and registries ranging from 20 to 3,251 tools. We then turn the same principle into a reinforcement learning (RL) reward for choosing tool shortlist depth per query. The RL agent is deliberately simple, serving as a probe of the metric rather than a proposed system. As the shortlist grows, random chance of including the correct tool rises, so the reward naturally decreases, reducing the need for an engineered depth penalty. On BFCL (370 tools), the learned policy nearly matches the coverage of showing 50 tools ($90.3\%$ vs $90.8\%$) while presenting only 7 on average. On ToolBench (3,251 tools), a fixed shortlist of 5 tools achieves higher aggregate coverage ($64.7\%$ vs $61.9\%$) but finds nothing on hard queries (correct tool ranked 6th-20th). The BoR agent finds $16.7\%$ on those same queries by searching deeper. Downstream validation with Claude Sonnet 4.6 indicates that shorter adaptive lists also improve the LLM's ability to select the right tool: $93.1\%$ versus $87.1\%$ when always shown 5 tools, widening to $76.8\%$ vs $60.9\%$ on medium-difficulty queries where the correct tool is present but not ranked first.

翻译：在大语言模型智能体使用工具之前，检索系统必须决定向智能体展示哪些候选工具。这份候选列表应该有多长？展示太多工具会导致模型难以选择；展示太少则可能遗漏正确工具。多数系统对每个查询采用固定的候选列表长度，但目前尚无标准指标来评估该长度是否恰当。本文将展示给大语言模型智能体的工具数量作为评估对象，并应用基于机会校正的指标“随机化超越比特数”（Bits-over-Random, BoR），该指标衡量给定深度下的成功率是否优于同一深度下的随机选择结果。我们在三个工具选择基准、多个评分器以及工具数量从20到3,251不等的工具库上评估了BoR。随后，我们将相同原理转化为用于选择每个查询的工具候选列表深度的强化学习奖励。该强化学习智能体被刻意设计得简单，作为指标探测工具而非建议系统。随着候选列表增长，包含正确工具的随机概率上升，奖励自然递减，从而减少了对人工设计深度惩罚项的需求。在BFCL（370个工具）上，学习到的策略几乎达到了展示50个工具的覆盖率（90.3%对比90.8%），而平均仅展示7个工具。在ToolBench（3,251个工具）上，固定展示5个工具虽能获得更高的总覆盖率（64.7%对比61.9%），但在困难查询（正确工具排名第6至20位）上却一无所获。通过加深搜索，基于BoR的智能体在相同查询上取得了16.7%的覆盖率。使用Claude Sonnet 4.6进行的下游验证表明，较短的适应性列表也能提升大模型选择正确工具的能力：始终展示5个工具时正确率为87.1%，而适应性列表达到93.1%；在正确工具存在但非排名第一的中等难度查询上，该差距进一步扩大至60.9%对比76.8%。