Large language models (LLMs) are increasingly deployed in cost-sensitive and on-device scenarios, and safety guardrails have advanced mainly in English. However, real-world Chinese malicious queries typically conceal intent via homophones, pinyin, symbol-based splitting, and other Chinese-specific patterns. These Chinese-specific adversarial patterns create the safety evaluation gap that is not well captured by existing benchmarks focused on English. This gap is particularly concerning for lightweight models, which may be more vulnerable to such specific adversarial perturbations. To bridge this gap, we introduce the Chinese-Specific Safety Benchmark (CSSBench) that emphasizes these adversarial patterns and evaluates the safety of lightweight LLMs in Chinese. Our benchmark covers six domains that are common in real Chinese scenarios, including illegal activities and compliance, privacy leakage, health and medical misinformation, fraud and hate, adult content, and public and political safety, and organizes queries into multiple task types. We evaluate a set of popular lightweight LLMs and measure over-refusal behavior to assess safety-induced performance degradation. Our results show that the Chinese-specific adversarial pattern is a critical challenge for lightweight LLMs. This benchmark offers a comprehensive evaluation of LLM safety in Chinese, assisting robust deployments in practice.

翻译：大型语言模型（LLM）正越来越多地部署在成本敏感和端侧场景中，而安全防护措施的发展主要集中于英语。然而，现实世界中的中文恶意查询通常通过同音字、拼音、基于符号的拆分以及其他中文特定模式来隐藏意图。这些中文特定的对抗模式造成了现有专注于英语的基准测试未能充分捕捉的安全评估差距。这一差距对于轻量级模型尤为令人担忧，因为它们可能更容易受到此类特定对抗性扰动的影响。为弥补这一差距，我们提出了中文特定安全基准（CSSBench），该基准强调这些对抗模式，并评估轻量级LLM在中文环境下的安全性。我们的基准覆盖了真实中文场景中常见的六个领域，包括非法活动与合规、隐私泄露、健康与医疗虚假信息、欺诈与仇恨、成人内容以及公共与政治安全，并将查询组织为多种任务类型。我们评估了一系列流行的轻量级LLM，并通过测量过度拒绝行为来评估因安全考虑导致的性能下降。我们的结果表明，中文特定对抗模式是轻量级LLM面临的关键挑战。该基准为LLM在中文环境下的安全性提供了全面评估，有助于在实际中实现稳健的部署。