Local prediction-error-based curiosity rewards focus on the current transition without considering the world model's cumulative prediction error across all visited transitions. We introduce Curiosity-Critic, which grounds its intrinsic reward in the improvement of this cumulative objective, and show that it admits a tractable per-step surrogate: the difference between the current prediction error and the asymptotic error baseline of the current state transition. We estimate this error baseline online with a learned critic co-trained alongside the world model; since the critic only has to learn how hard a transition is to predict, its estimate of the irreducible noise floor converges well before the world model saturates, redirecting exploration toward learnable transitions. The reward is higher for learnable transitions and collapses toward zero for stochastic ones, thereby separating epistemic (reducible) from aleatoric (irreducible) prediction error online. Prior prediction-error curiosity formulations, from Schmidhuber (1991) to learned-feature-space variants, emerge as special cases corresponding to specific approximations of this error baseline. Experiments on a stochastic grid world show that Curiosity-Critic outperforms prediction-error, visitation-count, and Random Network Distillation methods in training speed and final world model accuracy.

翻译：基于局部预测误差的好奇心奖励仅关注当前转移，而未考虑世界模型在所有已访问转移中的累积预测误差。我们提出“好奇-批评”（Curiosity-Critic），将其内在奖励建立在该累积目标的改进上，并证明它可得到一个可处理的每步替代量：当前预测误差与当前状态转移的渐近误差基准之差。我们通过一个与世界模型联合训练的学习批评者（critic）在线估计该误差基准；由于批评者只需学习转移的可预测难度，其对不可约噪声底线的估计在世界模型饱和前便已收敛，从而将探索导向可学习转移。该奖励对可学习转移较高，而对随机转移趋近于零，从而在线分离认知（可约简）预测误差与偶然（不可约简）预测误差。从Schmidhuber（1991）到学习特征空间变体的先前预测误差好奇心公式，均可作为该误差基准特定近似下的特例。在随机网格世界上的实验表明，“好奇-批评”在训练速度和最终世界模型精度上均优于基于预测误差、访问计数以及随机网络蒸馏的方法。