Large Language Models and Multi-Modal LLMs have become pervasive, and so does the importance of their security; yet, modern LLMs are known to be vulnerable to jailbreaking attacks. These attacks can allow malicious users to exploit the models, making the case for effective jailbreak detection mechanisms an essential aspect of maintaining the integrity and trustworthiness of LLM-based applications. However, existing detection works on jailbreak attacks have limitations. Existing post-query-based strategies require target domain knowledge, and pre-query-based methods mainly focus on text-level attacks and fail to meet the increasingly complex multi-modal security requirements placed upon contemporary LLMs. This gap underscores the need for a more comprehensive approach to safeguarding these influential systems. In this work, we propose JailGuard, the first mutation-based jailbreaking detection framework which supports both image and text modalities. Our key observation is that attack queries inherently possess less robustness compared to benign queries. Specifically, to confuse the model, attack queries are usually crafted with well-designed templates or complicate perturbations, leading to a fact that a slight disturbance in input may result in a drastic change in the response. This lack of robustness can be utilized in attack detection. Based on this intuition, we designed and implemented a detection framework comprising 19 different mutators and a divergence-based detection formula. To fully understand the effectiveness of our framework, we built the first multi-modal LLM jailbreaking attack dataset, which has 304 items of data, covering ten types of known jailbreaking attacks on image and text modalities. The evaluation suggests that JailGuard achieves the best detection accuracy of 89.38%/85.42% on image and text inputs, outperforming state-of-the-art defense methods by 15.28%.

翻译：大型语言模型和多模态大语言模型已变得无处不在，其安全性也日益重要。然而，现代大语言模型已知易受越狱攻击。这类攻击可能被恶意用户利用以操纵模型，这使得有效的越狱检测机制成为维护基于大语言模型应用完整性及可信度的关键。现有越狱攻击检测研究存在局限性：基于查询后处理的策略需要目标领域知识，而基于查询前处理的方法主要聚焦于文本级攻击，难以满足当代大语言模型日益复杂的多模态安全需求。这一空白凸显了需要更全面的方案来保障这些具有影响力的系统。本文提出JailGuard——首个支持图像和文本模态的基于突变的越狱检测框架。我们的关键发现是：攻击查询本质上比良性查询具有更低的鲁棒性。具体而言，为迷惑模型，攻击查询通常采用精心设计的模板或复杂扰动，导致输入微小变化可能引发响应剧烈变化。这种鲁棒性缺失可用于攻击检测。基于此直觉，我们设计并实现了包含19种不同变异器和基于差异度的检测公式的检测框架。为充分验证框架有效性，我们构建了首个多模态大语言模型越狱攻击数据集，包含304条数据，覆盖图像和文本模态的十种已知越狱攻击类型。评估表明，JailGuard在图像和文本输入上分别达到89.38%/85.42%的最佳检测准确率，较现有最优防御方法提升15.28%。