LLMs have shown impressive few-shot performance across many tasks. However, they still struggle when it comes to generating complex output structures, such as those required for Information Extraction. This limitation stems from the fact that LLMs, without finetuning, tend to generate free text rather than precise structures that follow a specific grammar. In this work, we propose to enrich the decoding step with formal grammar constraints. During beam search, only valid token continuations compliant with the grammar production rules are considered. This enforces the generation of valid sequences exclusively. Our framework is highly general and flexible, allowing any Context-Free Grammar (CFG) to be integrated into our custom constrained beam search implementation. We demonstrate that the outputs of many NLP tasks can be represented as formal languages, making them suitable for direct use in our framework. For task where the output space is dependent on the input, we propose input-dependent grammars to constrain the generation. We conducted experiments with two challenging tasks involving large alphabets in their grammar (Wikidata entities and relations): information extraction and entity disambiguation. Our results with LLaMA models clearly indicate that grammar-constrained decoding outperforms few-shot prompting without constraints, and even competes with task-specific finetuned models. These findings suggest that integrating grammar-based constraints during decoding holds great promise in making LLMs reliably produce structured outputs, especially in setting where training data is scarce and finetuning is expensive.

翻译：大语言模型（LLM）在多项任务中展现出卓越的少样本学习能力。然而，当需要生成信息提取等复杂输出结构时，它们仍存在明显不足。这一局限源于未经过微调的LLM倾向于生成自由文本，而非遵循特定语法的精确结构。本研究提出在解码阶段引入形式语法约束：在束搜索过程中，仅考虑符合语法产生式规则的有效标记延续方案，从而强制生成完全有效的序列。我们的框架具有高度通用性与灵活性，支持将任意上下文无关文法（CFG）集成至自定义的约束束搜索实现中。我们证明，诸多自然语言处理任务的输出均可表示为形式语言，因此可直接应用于本框架。针对输出空间依赖输入的任务，我们提出输入相关语法来约束生成过程。我们使用包含大规模字母表（Wikidata实体与关系）的语法，在信息提取与实体消歧两项挑战性任务上开展实验。基于LLaMA模型的实验结果表明，语法约束解码显著优于无约束的少样本提示方法，甚至可与任务特定微调模型相媲美。这些发现表明，在解码过程中整合基于语法的约束，对于使LLM可靠生成结构化输出具有重要潜力，尤其在训练数据稀缺且微调成本高昂的场景中。