We present and evaluate a method called grammar masking, which is used to guide large language models (LLMs) toward producing syntactically correct models for a given context-free grammar. Prompt engineering methods such as few-shot learning or priming can be used to improve the chances of an LLM producing correct syntax, but the more complex the grammar, the more time-consuming and less promising these methods become. Previous work is focused primarily on the usage of either language model training or prompt engineering. In this work, a method is presented that restricts the output to a given grammar using constrained decoding to ensure the output adheres to a valid syntax. We use several DSLs built with MontiCore and task multiple LLMs to produce models with and without constrained decoding. A corresponding parser is used to confirm the syntactic correctness of each model. We show that grammar masking can dramatically improve the modeling capabilities of several LLMs, reducing the need for well-refined prompting while increasing the chance of producing correct models.

翻译：本文提出并评估了一种称为语法掩码的方法，该方法用于引导大语言模型（LLMs）为给定的上下文无关文法生成句法正确的模型。虽然少样本学习或提示工程等方法可用于提高LLM生成正确语法的概率，但随着文法复杂度的增加，这些方法往往变得更加耗时且效果有限。先前的研究主要集中于语言模型训练或提示工程的应用。本工作提出了一种利用约束解码来限制输出符合特定文法的方法，从而确保输出遵循有效的语法结构。我们使用多个基于MontiCore构建的领域特定语言（DSLs），并让多个LLM在启用和未启用约束解码的条件下生成模型。通过相应的解析器验证每个模型的句法正确性。实验表明，语法掩码能显著提升多种LLM的建模能力，在降低对精细提示依赖的同时，大幅提高生成正确模型的概率。