Recent advancements in large language models (LLMs) have exhibited promising performance in solving sequential decision-making problems. By imitating few-shot examples provided in the prompts (i.e., in-context learning), an LLM agent can interact with an external environment and complete given tasks without additional training. However, such few-shot examples are often insufficient to generate high-quality solutions for complex and long-horizon tasks, while the limited context length cannot consume larger-scale demonstrations. To this end, we propose an offline learning framework that utilizes offline data at scale (e.g, logs of human interactions) to facilitate the in-context learning performance of LLM agents. We formally define LLM-powered policies with both text-based approaches and code-based approaches. We then introduce an Offline Data-driven Discovery and Distillation (O3D) framework to improve LLM-powered policies without finetuning. O3D automatically discovers reusable skills and distills generalizable knowledge across multiple tasks based on offline interaction data, advancing the capability of solving downstream tasks. Empirical results under two interactive decision-making benchmarks (ALFWorld and WebShop) demonstrate that O3D can notably enhance the decision-making capabilities of LLMs through the offline discovery and distillation process, and consistently outperform baselines across various LLMs with both text-based-policy and code-based-policy.

翻译：近年来，大规模语言模型（LLM）在解决序列决策问题方面展现出令人瞩目的性能。通过模仿提示中提供的少样本示例（即上下文学习），LLM智能体能够与外部环境交互并在无需额外训练的情况下完成给定任务。然而，对于复杂且长时域的任务，这类少样本示例往往不足以生成高质量解决方案，同时有限的上下文长度也无法容纳更大规模的示范。为此，我们提出一种离线学习框架，利用大规模离线数据（例如人类交互日志）来提升LLM智能体的上下文学习性能。我们正式定义了基于文本方法与基于代码方法的LLM驱动策略，进而引入离线数据驱动发现与蒸馏（O3D）框架，通过无需微调的方式改进LLM驱动策略。O3D基于离线交互数据自动发现可重用技能，并跨多个任务蒸馏通用知识，从而增强解决下游任务的能力。在两个交互式决策基准（ALFWorld和WebShop）上的实验结果表明，O3D能够通过离线发现与蒸馏过程显著提升LLM的决策能力，并且在基于文本策略与基于代码策略的多种LLM模型上均持续优于基线方法。