Programmatic reinforcement learning (PRL) has been explored for representing policies through programs as a means to achieve interpretability and generalization. Despite promising outcomes, current state-of-the-art PRL methods are hindered by sample inefficiency, necessitating tens of millions of program-environment interactions. To tackle this challenge, we introduce a novel LLM-guided search framework (LLM-GS). Our key insight is to leverage the programming expertise and common sense reasoning of LLMs to enhance the efficiency of assumption-free, random-guessing search methods. We address the challenge of LLMs' inability to generate precise and grammatically correct programs in domain-specific languages (DSLs) by proposing a Pythonic-DSL strategy - an LLM is instructed to initially generate Python codes and then convert them into DSL programs. To further optimize the LLM-generated programs, we develop a search algorithm named Scheduled Hill Climbing, designed to efficiently explore the programmatic search space to consistently improve the programs. Experimental results in the Karel domain demonstrate the superior effectiveness and efficiency of our LLM-GS framework. Extensive ablation studies further verify the critical role of our Pythonic-DSL strategy and Scheduled Hill Climbing algorithm.

翻译：程序化强化学习（PRL）通过程序表示策略，以实现可解释性和泛化能力，这一方向已得到广泛探索。尽管取得了有前景的成果，当前最先进的PRL方法仍受限于样本效率低下，往往需要数千万次的程序-环境交互。为应对这一挑战，我们提出了一种新颖的大语言模型引导搜索框架（LLM-GS）。我们的核心思路是利用大语言模型的编程专业知识与常识推理能力，提升无假设随机猜测搜索方法的效率。针对大语言模型难以生成符合领域特定语言（DSL）语法规范且精确的程序这一难题，我们提出了Python化DSL策略——指导大语言模型先生成Python代码，再将其转换为DSL程序。为进一步优化大语言模型生成的程序，我们开发了一种名为“计划性爬山法”的搜索算法，旨在高效探索程序化搜索空间，持续改进程序质量。在Karel领域中的实验结果表明，我们的LLM-GS框架具有卓越的有效性与效率。大量消融实验进一步验证了Python化DSL策略与计划性爬山法算法的关键作用。