Large Language Models and Multi-Modal LLMs have become pervasive, and so does the importance of their security; yet, modern LLMs are known to be vulnerable to jailbreaking attacks. These attacks can allow malicious users to exploit the models, making the case for effective jailbreak detection mechanisms an essential aspect of maintaining the integrity and trustworthiness of LLM-based applications. However, existing detection works on jailbreak attacks have limitations. Existing post-query-based strategies require target domain knowledge, and pre-query-based methods mainly focus on text-level attacks and fail to meet the increasingly complex multi-modal security requirements placed upon contemporary LLMs. This gap underscores the need for a more comprehensive approach to safeguarding these influential systems. In this work, we propose JailGuard, the first mutation-based jailbreaking detection framework which supports both image and text modalities. Our key observation is that attack queries inherently possess less robustness compared to benign queries. Specifically, to confuse the model, attack queries are usually crafted with well-designed templates or complicate perturbations, leading to a fact that a slight disturbance in input may result in a drastic change in the response. This lack of robustness can be utilized in attack detection. Based on this intuition, we designed and implemented a detection framework comprising 19 different mutators and a divergence-based detection formula. To fully understand the effectiveness of our framework, we built the first multi-modal LLM jailbreaking attack dataset, which has 304 items of data, covering ten types of known jailbreaking attacks on image and text modalities. The evaluation suggests that JailGuard achieves the best detection accuracy of 89.38%/85.42% on image and text inputs, outperforming state-of-the-art defense methods by 15.28%.

翻译：大型语言模型和多模态大语言模型已变得无处不在，其安全性也日益重要；然而，现代大语言模型已知容易受到越狱攻击。这些攻击可使恶意用户利用模型，因此有效的越狱检测机制成为维护基于大语言模型应用完整性和可信度的关键。然而，现有针对越狱攻击的检测工作存在局限性。基于查询后处理的策略需要目标域知识，而基于查询前处理的方法主要关注文本级攻击，无法满足当代大语言模型日益复杂的多模态安全需求。这一差距凸显了采用更全面的方法来保护这些有影响力系统的必要性。在这项工作中，我们提出了JailGuard，这是首个支持图像和文本模态的基于突变的越狱检测框架。我们的关键观察是：攻击查询本质上比良性查询具有更低的鲁棒性。具体而言，为迷惑模型，攻击查询通常利用精心设计的模板或复杂扰动构建，导致输入的微小扰动可能引发响应的剧烈变化。这种鲁棒性缺失可用于攻击检测。基于这一直觉，我们设计并实现了一个包含19种不同变异器及基于发散度的检测公式的检测框架。为全面评估框架有效性，我们构建了首个多模态大语言模型越狱攻击数据集，包含304条数据，覆盖图像和文本模态的十种已知越狱攻击。评估表明，JailGuard在图像和文本输入上分别达到89.38%和85.42%的最佳检测准确率，较最先进的防御方法提升15.28%。