LLMs have shown impressive few-shot performance across many tasks. However, they still struggle when it comes to reliably generating complex output structures, such as those required for information extraction. This limitation stems from the fact that LLMs, without fine-tuning, tend to generate free text rather than structures precisely following a specific grammar. In this work, we propose to enrich the decoding with formal grammar constraints. More concretely, given Context-Free Grammar(CFG), our framework ensures that the token generated in each decoding step would lead to a valid continuation compliant with the grammar production rules. This process guarantees the generation of valid sequences. Importantly, our framework can be readily combined with any CFG or decoding algorithm. We demonstrate that the outputs of many NLP tasks can be represented as formal languages, making them suitable for direct use in our framework. We conducted experiments with two challenging tasks involving large alphabets in their grammar (Wikidata entities and relations): information extraction and entity disambiguation. Our results with LLaMA models indicate that grammar-constrained decoding substantially outperforms unconstrained decoding and even competes with task-specific fine-tuned models. These findings suggest that integrating grammar-based constraints during decoding holds great promise in making LLMs reliably produce structured outputs, especially in setting where training data is scarce and fine-tuning is expensive.

翻译：大语言模型在许多任务中展现出令人印象深刻的少样本性能。然而，在可靠生成复杂输出结构（如信息提取所需的结构）方面仍存在困难。这一局限源于未经微调的大语言模型倾向于生成自由文本，而非精确遵循特定语法的结构。在本文中，我们提出通过形式语法约束来丰富解码过程。具体而言，给定上下文无关文法，我们的框架确保每个解码步骤生成的token能够产生符合语法产生式规则的有效后续内容。这一过程保证了有效序列的生成。重要的是，我们的框架可以便捷地与任何上下文无关文法或解码算法结合。我们证明了许多自然语言处理任务的输出可以表示为形式语言，使其适用于该框架。我们针对涉及大规模字母表（维基数据实体与关系）的两项具有挑战性的任务进行了实验：信息提取与实体消歧。在LLaMA模型上的结果表明，语法约束解码显著优于无约束解码，甚至能与特定任务微调模型相媲美。这些发现表明，在解码过程中融合基于语法的约束在使大语言模型可靠生成结构化输出方面具有巨大潜力，尤其是在训练数据稀缺且微调成本高昂的场景中。