Recent advancements in large language models (LLMs) have exhibited promising performance in solving sequential decision-making problems. By imitating few-shot examples provided in the prompts (i.e., in-context learning), an LLM agent can interact with an external environment and complete given tasks without additional training. However, such few-shot examples are often insufficient to generate high-quality solutions for complex and long-horizon tasks, while the limited context length cannot consume larger-scale demonstrations. To this end, we propose an offline learning framework that utilizes offline data at scale (e.g, logs of human interactions) to facilitate the in-context learning performance of LLM agents. We formally define LLM-powered policies with both text-based approaches and code-based approaches. We then introduce an Offline Data-driven Discovery and Distillation (O3D) framework to improve LLM-powered policies without finetuning. O3D automatically discovers reusable skills and distills generalizable knowledge across multiple tasks based on offline interaction data, advancing the capability of solving downstream tasks. Empirical results under two interactive decision-making benchmarks (ALFWorld and WebShop) demonstrate that O3D can notably enhance the decision-making capabilities of LLMs through the offline discovery and distillation process, and consistently outperform baselines across various LLMs with both text-based-policy and code-based-policy.

翻译：近期大语言模型（LLMs）在解决序列决策问题中展现出令人瞩目的性能。通过模仿提示中提供的少样本示例（即上下文学习），LLM智能体无需额外训练即可与外部环境交互并完成指定任务。然而，此类少样本示例往往难以生成复杂长程任务的高质量解决方案，而有限的上下文长度无法容纳更大规模的示范数据。为此，我们提出一种利用大规模离线数据（如人类交互日志）增强LLM智能体上下文学习性能的离线学习框架。我们通过基于文本和基于代码两种方法正式定义了LLM驱动的策略。随后引入离线数据驱动的发现与蒸馏（O3D）框架，无需微调即可提升LLM驱动策略的性能。O3D基于离线交互数据自动发现可复用技能并跨任务蒸馏通用知识，从而提升解决下游任务的能力。在两个交互式决策基准（ALFWorld和WebShop）上的实验结果表明，O3D通过离线发现与蒸馏过程显著增强了LLMs的决策能力，并在多种LLMs的文本策略与代码策略上持续超越基线方法。