Many structured prediction and reasoning tasks can be framed as program synthesis problems, where the goal is to generate a program in a domain-specific language (DSL) that transforms input data into the desired output. Unfortunately, purely neural approaches, such as large language models (LLMs), often fail to produce fully correct programs in unfamiliar DSLs, while purely symbolic methods based on combinatorial search scale poorly to complex problems. Motivated by these limitations, we introduce a hybrid approach, where LLM completions for a given task are used to learn a task-specific, context-free surrogate model, which is then used to guide program synthesis. We evaluate this hybrid approach on three domains, and show that it outperforms both unguided search and direct sampling from LLMs, as well as existing program synthesizers.

翻译：许多结构化预测和推理任务可被形式化为程序合成问题，其目标在于生成一种领域特定语言（DSL）中的程序，以将输入数据转换为期望输出。然而，纯神经方法（例如大型语言模型LLMs）在面对陌生DSL时往往无法生成完全正确的程序，而基于组合搜索的纯符号方法在处理复杂问题时扩展性较差。基于这些局限性，我们提出一种混合方法：首先利用LLM对给定任务生成的补全结果来学习一个任务特定的上下文无关代理模型，随后使用该模型指导程序合成。我们在三个领域对该混合方法进行评估，结果表明其性能优于无引导搜索、LLM直接采样以及现有程序合成器。