Large language models (LLMs) often fail to maintain safety in low-resource language varieties, such as code-mixed vernaculars and regional dialects. We introduce RabakBench, a multilingual safety benchmark and scalable pipeline localized to Singapore's unique linguistic landscape, covering Singlish, Chinese, Malay, and Tamil. We construct the benchmark through a three-stage pipeline: (1) Generate: augmenting real-world unsafe web content via LLM-driven red teaming; (2) Label: applying semi-automated multi-label annotation using majority-voted LLM labelers; and (3) Translate: performing high-fidelity, toxicity-preserving translation. The resulting dataset contains over 5,000 examples across six fine-grained safety categories. Despite using LLMs for scalability, our framework maintains rigorous human oversight, achieving 0.70-0.80 inter-annotator agreement. Evaluations of 13 state-of-the-art guardrails reveal significant performance degradation, underscoring the need for localized evaluation. RabakBench provides a reproducible framework for building safety benchmarks in underserved communities.

翻译：大型语言模型（LLMs）在低资源语言变体（如语码混合的方言和区域变体）中往往难以维持安全性。本文介绍RabakBench——一个针对新加坡独特语言景观（涵盖新加坡式英语、中文、马来语和泰米尔语）进行本地化的多语言安全基准测试框架及可扩展流程。我们通过三阶段流程构建该基准：（1）生成：通过LLM驱动的红队测试增强现实世界不安全网络内容；（2）标注：采用多数投票的LLM标注器进行半自动化多标签标注；（3）翻译：执行高保真且保持毒性特征的翻译。最终数据集包含六个细粒度安全类别共计5000余条样本。尽管使用LLMs以实现可扩展性，我们的框架仍保持严格的人工监督，实现了0.70-0.80的标注者间一致性。对13个前沿防护机制的评估显示出显著的性能衰减，凸显了本地化评估的必要性。RabakBench为服务不足社区构建安全基准提供了可复现的框架。