Scaling action-controllable world models is limited by the scarcity of action labels. While latent action learning promises to extract control interfaces from unlabeled video, learned latents often fail to transfer across contexts: they entangle scene-specific cues and lack a shared coordinate system. This occurs because standard objectives operate only within each clip, providing no mechanism to align action semantics across contexts. Our key insight is that although actions are unobserved, their semantic effects are observable and can serve as a shared reference. We introduce Seq$Δ$-REPA, a sequence-level control-effect alignment objective that anchors integrated latent action to temporal feature differences from a frozen, self-supervised video encoder. Building on this, we present Olaf-World, a pipeline that pretrains action-conditioned video world models from large-scale passive video. Extensive experiments demonstrate that our method learns a more structured latent action space, leading to stronger zero-shot action transfer and more data-efficient adaptation to new control interfaces than state-of-the-art baselines.

翻译：扩展动作可控世界模型受到动作标签稀缺性的限制。虽然潜在动作学习有望从未标注视频中提取控制接口，但学习到的潜在表示往往难以跨上下文迁移：它们与场景特定线索纠缠，且缺乏共享坐标系。这是由于标准目标函数仅在单个视频片段内操作，未提供跨上下文对齐动作语义的机制。我们的核心洞见是：尽管动作本身未被观测，但其语义效应是可观测的，可作为共享参照基准。我们提出Seq$Δ$-REPA——一种序列级控制效应对齐目标，通过冻结的自监督视频编码器生成的时间特征差异来锚定集成潜在动作。在此基础上，我们构建了Olaf-World：一个从大规模被动视频中预训练动作条件视频世界模型的完整流程。大量实验表明，我们的方法能学习到更具结构化的潜在动作空间，在零样本动作迁移和新控制接口的数据高效适应方面，均优于当前最先进的基线方法。