We present KidsNanny, a two-stage multimodal content moderation architecture for child safety. Stage 1 combines a vision transformer (ViT) with an object detector for visual screening (11.7 ms); outputs are routed as text not raw pixels to Stage 2, which applies OCR and a text based 7B language model for contextual reasoning (120 ms total pipeline). We evaluate on the UnsafeBench Sexual category (1,054 images) under two regimes: vision-only, isolating Stage 1, and multimodal, evaluating the full Stage 1+2 pipeline. Stage 1 achieves 80.27% accuracy and 85.39% F1 at 11.7 ms; vision-only baselines range from 59.01% to 77.04% accuracy. The full pipeline achieves 81.40% accuracy and 86.16% F1 at 120 ms, compared to ShieldGemma-2 (64.80% accuracy, 1,136 ms) and LlavaGuard (80.36% accuracy, 4,138 ms). To evaluate text-awareness, we filter two subsets: a text+visual subset (257 images) and a text-only subset (44 images where safety depends primarily on embedded text). On text-only images, KidsNanny achieves 100% recall (25/25 positives; small sample) and 75.76% precision; ShieldGemma-2 achieves 84% recall and 60% precision at 1,136 ms. Results suggest that dedicated OCR-based reasoning may offer recall-precision advantages on text-embedded threats at lower latency, though the small text-only subset limits generalizability. By documenting this architecture and evaluation methodology, we aim to contribute to the broader research effort on efficient multimodal content moderation for child safety.
翻译:我们提出了KidsNanny,一种用于儿童安全保护的两阶段多模态内容审核架构。第一阶段结合视觉Transformer(ViT)与目标检测器进行视觉筛查(11.7毫秒);其输出以文本形式(而非原始像素)路由至第二阶段,该阶段应用OCR和基于文本的7B语言模型进行上下文推理(完整管道总耗时120毫秒)。我们在UnsafeBench Sexual类别(1,054张图像)上评估了两种模式:纯视觉模式(仅隔离第一阶段)和多模态模式(评估完整的第1+2阶段管道)。第一阶段在11.7毫秒内实现了80.27%的准确率和85.39%的F1分数;纯视觉基线模型的准确率范围在59.01%至77.04%之间。完整管道在120毫秒内实现了81.40%的准确率和86.16%的F1分数,相比之下,ShieldGemma-2(准确率64.80%,耗时1,136毫秒)和LlavaGuard(准确率80.36%,耗时4,138毫秒)。为评估文本感知能力,我们筛选了两个子集:文本+视觉子集(257张图像)和纯文本子集(44张图像,其安全性主要取决于嵌入文本)。在纯文本图像上,KidsNanny实现了100%的召回率(25/25个正例;小样本)和75.76%的精确率;ShieldGemma-2在1,136毫秒内实现了84%的召回率和60%的精确率。结果表明,尽管纯文本子集规模较小限制了普适性结论,但基于OCR的专用推理可能在处理文本嵌入威胁时,以更低延迟提供召回率-精确率优势。通过记录此架构和评估方法,我们旨在为儿童安全领域高效多模态内容审核的更广泛研究做出贡献。