We present and evaluate a method called grammar masking, which is used to guide large language models (LLMs) toward producing syntactically correct models for a given context-free grammar. Prompt engineering methods such as few-shot learning or priming can be used to improve the chances of an LLM producing correct syntax, but the more complex the grammar, the more time-consuming and less promising these methods become. Previous work is focused primarily on the usage of either language model training or prompt engineering. In this work, a method is presented that restricts the output to a given grammar using constrained decoding to ensure the output adheres to a valid syntax. We use several DSLs built with MontiCore and task multiple LLMs to produce models with and without constrained decoding. A corresponding parser is used to confirm the syntactic correctness of each model. We show that grammar masking can dramatically improve the modeling capabilities of several LLMs, reducing the need for well-refined prompting while increasing the chance of producing correct models.

翻译：本文提出并评估了一种名为语法掩码的方法，用于引导大语言模型针对给定的上下文无关文法生成句法正确的模型。尽管通过提示工程方法（如少样本学习或提示引导）可以提高LLM生成正确语法的概率，但随着文法复杂度的增加，这类方法往往耗时且效果有限。已有研究主要集中于语言模型训练或提示工程的应用。本研究提出一种利用约束解码限制输出符合特定文法的方法，从而确保输出具有有效的句法结构。我们使用基于MontiCore构建的多种领域特定语言，并让多个LLM在启用与未启用约束解码的条件下生成模型。通过对应解析器验证每个模型的句法正确性。实验表明，语法掩码能显著提升多个LLM的建模能力，在降低对精细提示依赖的同时，大幅提高生成正确模型的概率。