Current multimodal toxicity benchmarks typically use a single binary hatefulness label. This coarse approach conflates two fundamentally different characteristics of expression: tone and content. Drawing on communication science theory, we introduce a fine-grained annotation scheme that distinguishes two separable dimensions: incivility (rude or dismissive tone) and intolerance (content that attacks pluralism and targets groups or identities) and apply it to 2,030 memes from the Hateful Memes dataset. We evaluate different vision-language models under coarse-label training, transfer learning across label schemes and a joint learning approach that combines the coarse hatefulness label with our fine-grained annotations. Our results show that fine-grained annotations complement existing coarse labels and, when used jointly, improve overall model performance. Moreover, models trained with the fine-grained scheme exhibit more balanced moderation-relevant error profiles and are less prone to under-detection of harmful content than models trained on hatefulness labels alone (FNR-FPR, the difference between false negative and false positive rates: 0.74 to 0.42 for LLaVA-1.6-Mistral-7B; 0.54 to 0.28 for Qwen2.5-VL-7B). This work contributes to data-centric approaches in content moderation by improving the reliability and accuracy of moderation systems through enhanced data quality. Overall, combining both coarse and fine-grained labels provides a practical route to more reliable multimodal moderation.

翻译：当前的多模态毒性基准通常采用单一的二元仇恨标签。这种粗略方法混淆了表达的两个根本不同特征：语气与内容。借鉴传播科学理论，我们引入了一种细粒度的标注方案，区分了两个可分离的维度：不文明（粗鲁或轻蔑的语气）和不容忍（攻击多元性、针对群体或身份的内容），并将其应用于来自 Hateful Memes 数据集的 2030 个模因。我们在粗标签训练、跨标签方案迁移学习以及结合粗粒度仇恨标签与我们的细粒度标注的联合学习方法下，评估了不同的视觉语言模型。结果表明，细粒度标注补充了现有的粗标签，并在联合使用时提升了整体模型性能。此外，与仅使用仇恨标签训练的模型相比，采用细粒度方案训练的模型展现出更均衡的审查相关错误概况，且不易漏检有害内容（FNR-FPR，即假阴性与假阳性率之差：LLaVA-1.6-Mistral-7B 从 0.74 降至 0.42；Qwen2.5-VL-7B 从 0.54 降至 0.28）。这项工作通过提升数据质量，改进了审核系统的可靠性与准确性，从而为内容审核中数据驱动的方法做出了贡献。总体而言，结合粗标签与细粒度标签，为实现更可靠的多模态审核提供了一条实用路径。