Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: https://philip-mit.github.io/sole-r1/

翻译：视觉语言模型（VLM）在各类任务中展现出卓越能力，促使研究者尝试利用这些模型监督机器人学习。然而，当这些模型作为强化学习（RL）中的评估器时，当前最强的模型在部分可观测性和分布偏移条件下常会失效，导致策略利用感知错误而非真正完成任务。我们提出SOLE-R1（自观察学习者），这是一种专为在线RL提供唯一奖励信号而设计的视频语言推理模型。仅凭原始视频观测和自然语言目标，SOLE-R1即能执行逐时间步长的时空思维链（CoT）推理，并生成可直接用作奖励的密集任务进度估计。为训练SOLE-R1，我们开发了大规模视频轨迹与推理合成流水线，生成与连续进度监督对齐的时序化CoT轨迹。这些数据结合基础空间与多帧时序推理训练，采用融合监督微调与可验证奖励强化学习的混合框架进行模型训练。在四个不同仿真环境与真实机器人场景中，SOLE-R1实现从随机初始化的零样本在线RL：机器人无需真实奖励、成功指标、示范或任务特定调优即可学习前所未见的操作任务。SOLE-R1成功完成24个未见任务，显著优于包括Robometer、RoboReward、ReWiND、GPT-5和Gemini-3-Pro在内的强视觉语言奖励模型，同时对奖励破解展现出更强的鲁棒性。我们已在匿名页面https://philip-mit.github.io/sole-r1/ 公开所有模型、数据、代码与演示。