While reinforcement learning (RL) has empowered multi-turn reasoning agents with retrieval and tools, existing successes largely depend on extensive on-policy rollouts in high-cost, high-accuracy regimes. Under realistic resource constraints that cannot support large models or dense explorations, however, small language model agents fall into a low-cost, low-accuracy regime, where limited rollout budgets lead to sparse exploration, sparse credit assignment, and unstable training. In this work, we challenge this trade-off and show that small language models can achieve strong multi-hop reasoning under resource constraints. We introduce DAVID-GRPO, a budget-efficient RL framework that (i) stabilizes early learning with minimal supervision, (ii) assigns retrieval credit based on evidence recall, and (iii) improves exploration by resampling truncated near-miss trajectories. Evaluated on agents up to 1.5B parameters trained on only four RTX 3090 GPUs, DAVID-GRPO consistently outperforms prior RL methods designed for large-scale settings on six multi-hop QA benchmarks. These results show that with the right inductive biases, small agents can achieve low training cost with high accuracy.

翻译：尽管强化学习（RL）已通过检索与工具赋能了多轮推理智能体，但现有成果主要依赖于在高成本、高精度机制下进行大量在线策略推演。然而，在实际无法支持大型模型或密集探索的资源约束下，小型语言模型智能体会陷入低成本、低精度的困境：有限的推演预算导致稀疏探索、稀疏信用分配及训练不稳定。本研究挑战了这种权衡关系，证明小型语言模型在资源约束下仍可实现强大的多跳推理能力。我们提出DAVID-GRPO——一种预算高效的强化学习框架，其具备以下特性：（i）通过最小监督实现早期学习稳定化；（ii）基于证据召回机制分配检索信用；（iii）通过重采样截断近失轨迹提升探索效率。在仅使用四块RTX 3090 GPU训练、参数量不超过1.5B的智能体上进行评估，DAVID-GRPO在六个多跳问答基准测试中持续超越先前为大规模场景设计的强化学习方法。这些结果表明：通过恰当的归纳偏置，小型智能体能够以低训练成本实现高精度推理。