Learning from self-sampled data and sparse environmental feedback remains a fundamental challenge in training self-evolving agents. Temporal credit assignment mitigates this issue by transforming sparse feedback into dense supervision signals. However, previous approaches typically depend on learning task-specific value functions for credit assignment, which suffer from poor sample efficiency and limited generalization. In this work, we propose to leverage pretrained knowledge from large language models (LLMs) to transform sparse rewards into dense training signals (i.e., the advantage function) through retrospective in-context learning (RICL). We further propose an online learning framework, RICOL, which iteratively refines the policy based on the credit assignment results from RICL. We empirically demonstrate that RICL can accurately estimate the advantage function with limited samples and effectively identify critical states in the environment for temporal credit assignment. Extended evaluation on four BabyAI scenarios show that RICOL achieves comparable convergent performance with traditional online RL algorithms with significantly higher sample efficiency. Our findings highlight the potential of leveraging LLMs for temporal credit assignment, paving the way for more sample-efficient and generalizable RL paradigms.

翻译：从自采样数据和稀疏环境反馈中学习，仍然是训练自进化智能体的一个根本性挑战。时序信用分配通过将稀疏反馈转化为密集的监督信号来缓解这一问题。然而，先前的方法通常依赖于学习特定任务的价值函数来进行信用分配，这存在样本效率低下和泛化能力有限的问题。在本工作中，我们提出利用大语言模型（LLMs）的预训练知识，通过回顾式情境学习（RICL）将稀疏奖励转化为密集的训练信号（即优势函数）。我们进一步提出了一个在线学习框架RICOL，该框架基于RICL的信用分配结果迭代地优化策略。我们通过实验证明，RICL能够利用有限的样本准确估计优势函数，并有效识别环境中对时序信用分配至关重要的关键状态。在四个BabyAI场景上的扩展评估表明，RICOL在达到与传统在线强化学习算法相当的收敛性能的同时，具有显著更高的样本效率。我们的发现凸显了利用LLMs进行时序信用分配的潜力，为构建更具样本效率和泛化能力的强化学习范式铺平了道路。