Search agents powered by Large Language Models (LLMs) have demonstrated significant potential in tackling knowledge-intensive tasks. Reinforcement learning (RL) has emerged as a powerful paradigm for training these agents to perform complex, multi-step reasoning. However, prior RL-based methods often rely on sparse or rule-based rewards, which can lead agents to commit to suboptimal or erroneous reasoning paths without the ability to recover. To address these limitations, we propose ReSeek, a novel self-correcting framework for training search agents. Our framework introduces a self-correction mechanism that empowers the agent to dynamically identify and recover from erroneous search paths during an episode. By invoking a special JUDGE action, the agent can judge the information and re-plan its search strategy. To guide this process, we design a dense, instructive process reward function, which decomposes into a correctness reward for retrieving factual information and a utility reward for finding information genuinely useful for the query. Furthermore, to mitigate the risk of data contamination in existing datasets, we introduce FictionalHot, a new and challenging benchmark with recently curated questions requiring complex reasoning. Being intuitively reasonable and practically simple, extensive experiments show that agents trained with ReSeek significantly outperform SOTA baselines in task success rate and path faithfulness.

翻译：基于大语言模型（LLMs）的搜索智能体在处理知识密集型任务方面展现出巨大潜力。强化学习（RL）已成为训练这些智能体执行复杂多步推理的强大范式。然而，先前基于RL的方法通常依赖于稀疏或基于规则的奖励，这可能导致智能体陷入次优或错误的推理路径而无法恢复。为应对这些局限性，我们提出ReSeek，一种用于训练搜索智能体的新型自校正框架。该框架引入了一种自校正机制，使智能体能够在单次任务执行过程中动态识别并纠正错误的搜索路径。通过调用特殊的JUDGE动作，智能体可评估信息并重新规划其搜索策略。为引导这一过程，我们设计了一种密集的、具有指导性的过程奖励函数，该函数分解为用于检索事实信息的正确性奖励和用于查找对查询真正有用信息的效用奖励。此外，为减轻现有数据集中数据污染的风险，我们引入了FictionalHot——一个新颖且具有挑战性的基准测试集，包含近期收集的、需要复杂推理的问题。凭借其直观合理性和实践简洁性，大量实验表明，采用ReSeek训练的智能体在任务成功率和路径忠实度方面显著优于现有最先进的基线方法。