Co-evolutionary self-play, where one language model generates problems and another solves them, promises autonomous curriculum learning without human supervision. In practice, the proposer quickly converges to a narrow distribution of problems that satisfy the reward function. This diversity collapse renders the curriculum uninformative for the solver, stalling the co-evolutionary loop. We introduce vocabulary dropout, a random mask applied to the proposer's output logits during both policy training and curriculum generation, as a lightweight mechanism to sustain diversity. The mask is hard and non-stationary, preventing the proposer from locking into fixed token sequences. Training Qwen3-4B and Qwen3-8B on mathematical reasoning via R-Zero, we find that vocabulary dropout sustains proposer diversity across lexical, semantic, and functional metrics throughout training. It also yields solver improvements averaging +4.4 points at 8B, with the largest gains on competition-level benchmarks. Our findings suggest that explicit action-space constraints, analogous to the structural role that game rules play in classical self-play, can help sustain productive co-evolution in language. Vocabulary dropout is one simple instantiation of this principle.

翻译：协同进化自博弈——即一个语言模型生成问题，另一个模型求解问题——有望在无需人类监督的情况下实现自主课程学习。然而在实践中，问题生成模型会迅速收敛到满足奖励函数的狭窄问题分布。这种多样性崩溃导致课程对求解模型失去信息价值，使协同进化循环陷入停滞。我们提出词汇辍学（vocabulary dropout）方法——在策略训练和课程生成阶段对问题生成模型的输出对数概率施加随机掩码——作为一种维持多样性的轻量机制。该掩码具有硬性（hard）和非平稳性（non-stationary），可防止问题生成模型锁定固定的词元序列。通过R-Zero方法在数学推理任务上训练Qwen3-4B和Qwen3-8B模型，我们发现词汇辍学能在整个训练过程中持续维持问题生成模型在词汇、语义和功能指标上的多样性。该方法还使求解模型平均提升+4.4分（8B规模），在竞赛级基准测试上取得最大增益。我们的研究表明：显式的动作空间约束——类似于经典自博弈中游戏规则的结构性作用——有助于在语言领域维持富有成效的协同进化，而词汇辍学正是这一原理的简单实例化。