We present MineNPC-Task, a user-authored benchmark and evaluation harness for testing memory-aware, mixed-initiative LLM agents in open-world Minecraft. Rather than relying on synthetic prompts, tasks are elicited through formative and summative co-play with expert players, then normalized into parametric templates with explicit preconditions and dependency structure. These tasks are paired with machine-checkable validators under a bounded-knowledge policy that forbids out-of-world shortcuts. The harness captures plan, action, and memory events, including plan previews, targeted clarifications, memory reads and writes, precondition checks, and repair attempts, and reports outcomes relative to the total number of attempted subtasks using only in-world evidence. As an initial snapshot, we instantiate the framework with GPT-4o and evaluate 216 subtasks across 8 experienced players. We observe recurring breakdown patterns in code execution, inventory and tool handling, referencing, and navigation, alongside successful recoveries supported by mixed-initiative clarifications and lightweight memory use. Participants rated interaction quality and interface usability positively, while noting the need for stronger memory persistence across tasks. We release the complete task suite, validators, logs, and evaluation harness to support transparent and reproducible evaluation of future memory-aware embodied agents.

翻译：我们提出了MineNPC-Task，这是一个由用户创建的基准测试与评估框架，用于在开放世界游戏Minecraft中测试具有记忆感知能力的混合主动式LLM智能体。该任务集并非依赖合成提示生成，而是通过与资深玩家进行形成性和总结性协同游戏来激发设计，随后将任务规范化为具有明确前置条件和依赖结构的参数化模板。这些任务与机器可验证的校验器配对，并遵循禁止使用超世界捷径的有界知识策略。该框架捕获计划、行动和记忆事件，包括计划预览、针对性澄清、记忆读写、前置条件检查及修复尝试，并仅依据游戏内证据，根据已尝试子任务的总数报告结果。作为初始版本，我们使用GPT-4o实例化该框架，并对8名经验丰富的玩家进行了216项子任务评估。我们观察到在代码执行、物品栏与工具处理、对象引用及导航方面存在重复性故障模式，同时也发现了通过混合主动式澄清和轻量级记忆使用实现成功恢复的案例。参与者对交互质量和界面可用性给予积极评价，同时指出需要增强跨任务记忆持久性。我们公开了完整的任务套件、校验器、日志及评估框架，以支持未来记忆感知具身智能体的透明化与可复现评估。