Reinforcement Learning (RL) for Large Language Models (LLMs) faces a fundamental tension: the numerical divergence between high-throughput inference engines and numerically precise training engines. Although these systems share the same parameters, they produce slightly different probability distributions, creating a training-inference mismatch. We prove that the bound on the log-probability divergence arising from this mismatch scales as $(1-p)$, where $p$ is the token probability. This scaling induces a highly asymmetric effect: the bound vanishes for high-probability tokens but remains significant for low-probability tokens in the distribution tail. When sampled, these tail tokens introduce systematically biased errors that accumulate over sequences, thereby destabilizing gradient estimation. Instead of applying post-hoc corrections, we propose Dynamic Vocabulary Pruning (DVP), which constrains the RL objective to a dynamically determined ''safe'' vocabulary that excludes the extreme tail. This strategy trades large, destabilizing numerical errors for a small, bounded optimization bias. We validate DVP empirically by demonstrating stable training, and theoretically by deriving strict bounds on the induced bias.

翻译：大语言模型（LLM）的强化学习（RL）面临一个根本性矛盾：高吞吐量推理引擎与数值精确的训练引擎之间的数值发散性。尽管这些系统共享相同的参数，它们会产生略微不同的概率分布，从而造成训练-推理不匹配。我们证明，由这种不匹配产生的对数概率散度的上界按$(1-p)$缩放，其中$p$为词元概率。这种缩放导致高度不对称的效应：对于高概率词元，该上界趋于零，但对于分布尾部的低概率词元，该上界仍然显著。当对这些尾部词元进行采样时，它们会引入系统性偏差误差，这些误差在序列中不断累积，从而破坏梯度估计的稳定性。我们提出动态词汇剪枝（DVP）方法，而非采用事后校正。该方法将RL目标约束在一个动态确定的“安全”词汇集内，该集合排除了极端尾部词元。此策略以小的、有界的优化偏差替代了大的、破坏稳定性的数值误差。我们通过实验验证了DVP能实现稳定训练，并通过理论推导严格界定了其所引入的偏差。