While reinforcement learning (RL) successfully enhances reasoning in large language models, its role in fostering compositional generalization (the ability to synthesize novel skills from known components) is often conflated with mere length generalization. To this end, we study what RL post-training teaches about skill composition and how the structure of the composition affects the skill transfer. We focus on the Countdown task (given n numbers and a target, form an expression that evaluates to the target) and analyze model solutions as expression trees, where each subtree corresponds to a reusable subtask and thus can be viewed as a ``skill.'' Tracking tree shapes and their success rates over training, we find: (i) out-of-distribution (OOD) generalization to larger n and to unseen tree shapes, indicating compositional reuse of subtasks; (ii) a structure-dependent hierarchy of learnability -- models master shallow balanced trees (workload is balanced between subtasks) before deep unbalanced ones, with persistent fragility on right-heavy structures (even when the composition depth is the same as some left-heavy structures). Our diagnostic reveals what is learned, in what order, and where generalization fails, clarifying how RL-only post-training induces OOD generalization beyond what standard metrics such as pass@k reveal.

翻译：尽管强化学习（RL）成功提升了大型语言模型的推理能力，但其在促进组合泛化（即从已知组件合成新技能的能力）中的作用常被与单纯的长度泛化混为一谈。为此，我们研究RL后训练在技能组合方面教授了什么，以及组合结构如何影响技能迁移。我们聚焦于倒计时任务（给定n个数字和一个目标值，构建一个计算结果为目标值的表达式），并将模型解分析为表达式树，其中每个子树对应一个可复用的子任务，因而可视为一种“技能”。通过追踪训练过程中树形结构及其成功率，我们发现：（i）对更大n值及未见树形的分布外（OOD）泛化，表明子任务的组合式复用；（ii）一种结构依赖的学习难度层次——模型先掌握浅层平衡树（子任务间工作量均衡），后掌握深层非平衡树，且对右重结构（即使其组合深度与某些左重结构相同）表现出持续的脆弱性。我们的诊断揭示了所学内容、学习顺序及泛化失败之处，阐明了仅通过RL后训练如何诱导超越标准指标（如pass@k）所揭示的OOD泛化。