Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult. Effective policy extraction requires exploiting the critic's action gradient, yet directly backpropagating this signal through a multi-step denoising process can be numerically unstable. Existing methods work around this either by discarding gradient information, distilling the policy into a simpler one-step actor, or repeatedly fine-tuning the denoising policy as the critic improves. We propose QPILOTS, a method that leaves the original policy unmodified and steers the denoising process at inference time. At each denoising step, instead of evaluating the critic on the noisy intermediate action where critic predictions are unreliable, we first project that intermediate state to an estimate of the final clean action and compute the critic gradient there. We introduce two variants: QPILOTS-U uses a fast single-point approximation, while QPILOTS-M draws differentiable posterior samples via a learned auxiliary network. On a standard offline-to-online RL benchmark, QPILOTS achieves the best aggregate performance, reaching an average success rate of 90% across 50 tasks. We also apply QPILOTS to steer a large, frozen, pretrained Vision-Language Action (VLA) foundation model, outperforming or matching prior inference-time approaches across six manipulation tasks in simulation.

翻译：流匹配策略与扩散策略是表达能力强的动作生成器，但使用时序差分强化学习对其进行优化仍存在困难。有效策略提取需要利用评论家的动作梯度，然而直接通过多步去噪过程反向传播该信号可能导致数值不稳定。现有方法要么舍弃梯度信息、将策略蒸馏为更简单的单步执行器，要么随着评论家改进而反复微调去噪策略。我们提出QPILOTS方法，该方法保持原始策略不变，在推理阶段引导去噪过程。在每个去噪步骤中，我们首先将中间状态投影到最终干净动作的估计值，并在该处计算评论家梯度——而非在评论家预测不可靠的含噪中间动作上进行评估。我们引入两种变体：QPILOTS-U采用快速单点近似，而QPILOTS-M则通过学习的辅助网络抽取可微后验样本。在标准离线到在线强化学习基准测试中，QPILOTS实现了最佳总体性能，在50个任务上达到平均90%的成功率。我们还将QPILOTS应用于引导大规模冻结预训练视觉语言动作基础模型，在六项仿真操作任务中性能优于或持平于现有推理时方法。