Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications.

翻译：[translated abstract in Chinese] 大语言模型（LLMs）越来越多地被用于代码生成，引发了其可能被滥用以生成恶意代码的担忧。与此同时，语法约束解码（Grammar-Constrained Decoding，GCD）通过强制执行语法有效性，已被广泛采用以提升LLM生成代码的可靠性。本文揭示了一种反直觉的风险：这种面向可靠性的技术本身可能成为攻击面。我们提出了一种名为CodeSpear的新型越狱攻击，该攻击利用GCD诱导LLM生成恶意代码。实验表明，仅施加良性的代码语法约束便可有效越狱LLM。为应对这一漏洞，我们提出了CodeShield——一种即使在攻击者控制的语法约束下也能鲁棒地保持安全行为的安全对齐方法。CodeShield通过教导模型在GCD下生成蜜罐代码，在代码模态上对齐模型。此类代码在语义上无害（即不实现恶意请求），且在结构上多样化（使其难以通过语法收紧被抑制）。同时，CodeShield在自然语言可用时仍保留自然语言的拒绝机制。在涵盖4项基准测试的10个主流LLM上的实验表明，CodeSpear优于代表性越狱基线方法，平均攻击成功率提升超过30个百分点。CodeShield在CodeSpear攻击下恢复安全性的同时，保持了良性的实用性。我们的发现揭示了GCD的基础性风险，并呼吁对其潜在安全影响给予更多关注。