Constrained decoding, a technique for enforcing constraints on language model outputs, offers a way to control text generation without retraining or architectural modifications. Its application is, however, typically restricted to models that give users access to next-token distributions (usually via softmax logits), which poses a limitation with blackbox large language models (LLMs). This paper introduces sketch-guided constrained decoding (SGCD), a novel approach to constrained decoding for blackbox LLMs, which operates without access to the logits of the blackbox LLM. SGCD utilizes a locally hosted auxiliary model to refine the output of an unconstrained blackbox LLM, effectively treating this initial output as a "sketch" for further elaboration. This approach is complementary to traditional logit-based techniques and enables the application of constrained decoding in settings where full model transparency is unavailable. We demonstrate the efficacy of SGCD through experiments in closed information extraction and constituency parsing, showing how it enhances the utility and flexibility of blackbox LLMs for complex NLP tasks.

翻译：约束解码是一种对语言模型输出施加约束的技术，可在无需重新训练或修改模型架构的前提下控制文本生成过程。然而，该技术通常仅适用于允许用户访问下一词元分布（通常通过softmax对数几率）的模型，这限制了其在黑盒大语言模型（LLM）中的应用。本文提出草图引导的约束解码（SGCD），这是一种面向黑盒LLM的新型约束解码方法，无需访问黑盒LLM的对数几率。SGCD通过本地部署的辅助模型对未经约束的黑盒LLM输出进行精炼，将原始输出视为可进一步优化的"草图"。该方法与传统基于对数几率的约束解码技术形成互补，使约束解码能够在模型透明度不足的场景中应用。我们通过封闭信息抽取与成分句法分析实验验证了SGCD的有效性，证明其能显著增强黑盒LLM在复杂自然语言处理任务中的实用性与灵活性。