Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. To address this limitation, we introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including GPT-5 and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking.

翻译：视觉语言模型在各类任务中展现了卓越的能力，这促使研究者尝试利用这些模型监督机器人学习。然而，当这些模型作为强化学习中的评估器时，当前最先进的模型往往在部分可观测性和分布偏移条件下失效，导致策略利用感知误差而非真正完成任务。为解决这一局限，我们提出SOLE-R1（自观测学习器），一种专为在线强化学习设计、作为唯一奖励信号的视频语言推理模型。仅凭原始视频观测和自然语言目标，SOLE-R1执行逐时间步的时空链式推理，生成可直接用作奖励的密集任务进度估计。为训练SOLE-R1，我们开发了大规模视频轨迹与推理合成流程，生成与连续进度监督对齐的时域链式推理轨迹。这些数据结合基础空间推理与多帧时序推理能力，通过融合监督微调与可验证奖励强化学习的混合框架训练模型。在四个不同的仿真环境和一个真实机器人场景中，SOLE-R1实现了从随机初始化的零样本在线强化学习：机器人无需真实奖励、成功指标、示范或任务特定调参，即可学习先前未见过的操作任务。SOLE-R1成功完成24项未见任务，显著优于包括GPT-5和Gemini-3-Pro在内的强视觉语言奖励模型，同时对奖励篡改表现出更强的鲁棒性。