Traditional reinforcement-learning-based agents rely on sparse rewards that often only use binary values to indicate task completion or failure. The challenge in exploration efficiency makes it difficult to effectively learn complex tasks in Minecraft. To address this, this paper introduces an advanced learning system, named Auto MC-Reward, that leverages Large Language Models (LLMs) to automatically design dense reward functions, thereby enhancing the learning efficiency. Auto MC-Reward consists of three important components: Reward Designer, Reward Critic, and Trajectory Analyzer. Given the environment information and task descriptions, the Reward Designer first design the reward function by coding an executable Python function with predefined observation inputs. Then, our Reward Critic will be responsible for verifying the code, checking whether the code is self-consistent and free of syntax and semantic errors. Further, the Trajectory Analyzer summarizes possible failure causes and provides refinement suggestions according to collected trajectories. In the next round, Reward Designer will take further refine and iterate the dense reward function based on feedback. Experiments demonstrate a significant improvement in the success rate and learning efficiency of our agents in complex tasks in Minecraft, such as obtaining diamond with the efficient ability to avoid lava, and efficiently explore trees and animals that are sparse on the plains biome.

翻译：传统基于强化学习的智能体通常依赖稀疏奖励机制，仅通过二元值表示任务完成或失败状态。由于探索效率不足，这类方法难以在Minecraft中有效学习复杂任务。针对该问题，本文提出一种名为Auto MC-Reward的先进学习系统，利用大语言模型（LLMs）自动设计密集奖励函数，从而提升学习效率。Auto MC-Reward包含三个核心组件：奖励设计器、奖励评估器与轨迹分析器。首先，奖励设计器基于环境信息和任务描述，通过编写含预定义观测输入的可执行Python函数来设计奖励函数；随后，奖励评估器负责验证代码的一致性及语法语义正确性；轨迹分析器则根据收集的轨迹总结可能的失败原因并提供优化建议。在下一轮迭代中，奖励设计器将依据反馈进一步优化和迭代密集奖励函数。实验表明，本方法显著提升了智能体在Minecraft复杂任务中的成功率与学习效率，例如在高效规避熔岩的情况下获取钻石，以及高效探索平原生物群落中稀疏分布的树木与动物。