Constructing efficient and reliable policies to assist humans is indispensable for human-AI collaboration. Existing methods mainly follow two lines of work. Most prior work relies on multi-agent reinforcement learning (MARL) to learn black-box policies, which limits interpretability and raises safety concerns. Recent methods query large language models (LLMs) at each decision step, causing slow responses and high inference costs. We propose Collaboration Policy Tree (Co-pi-tree), a closed-loop method that learns an executable policy tree consisting of a partner-behavior prediction tree and an agent-action selection tree. Co-pi-tree constructs a policy by distilling LLM reasoning into policy tree code. It then evaluates the policy through partner interaction, obtains feedback, and uses natural language to summarize the interaction feedback to improve problematic branches. Experiments in Overcooked-AI show that Co-pi-tree improves average reward by 35.4% over the baseline average, while reducing the number of LLM queries by 77.7% and test-time latency by 97.1%. Project page: https://beiwenzhang.github.io/Co-pi-tree/

翻译：构建高效可靠的策略以辅助人类是人机协作中不可或缺的一环。现有方法主要遵循两条技术路线：早期工作多依赖多智能体强化学习（MARL）学习黑盒策略，这限制了可解释性并引发安全性担忧；近期方法则在每个决策步骤调用大语言模型（LLM），导致响应缓慢且推理成本高昂。我们提出协作策略树（Co-pi-tree），这是一种闭环方法，可学习由合作伙伴行为预测树与智能体动作选择树组成的可执行策略树。Co-pi-tree通过将LLM推理蒸馏为策略树代码来构建策略，随后通过伙伴交互评估策略、获取反馈，并利用自然语言总结交互反馈以改进问题分支。在Overcooked-AI上的实验表明：Co-pi-tree相比基线平均奖励提升35.4%，同时将LLM调用次数减少77.7%，测试时延降低97.1%。项目页面：https://beiwenzhang.github.io/Co-pi-tree/