Online hate remains a significant societal challenge, especially as multimodal content enables subtle, culturally grounded, and implicit forms of harm. Hateful memes embed hostility through text-image interactions and humor, making them difficult for automated systems to interpret. Although recent Vision-Language Models (VLMs) perform well on explicit cases, their deployment is limited by high inference costs and persistent failures on nuanced content. This work examines how far small models can be improved through prompt optimization, fine-tuning, and automated data augmentation. We introduce an end-to-end pipeline that varies prompt structure, label granularity, and training modality, showing that structured prompts and scaled supervision significantly strengthen compact VLMs. We also develop a multimodal augmentation framework that generates counterfactually neutral memes via a coordinated LLM-VLM setup, reducing spurious correlations and improving the detection of implicit hate. Ablation studies quantify the contribution of each component, demonstrating that prompt design, granular labels, and targeted augmentation collectively narrow the gap between small and large models. The results offer a practical path toward more robust and deployable multimodal hate-detection systems without relying on costly large-model inference.

翻译：在线仇恨内容仍然是重大的社会挑战，尤其是在多模态内容催生了微妙、文化相关且隐晦的伤害形式的情况下。仇恨性模因通过图文互动与幽默手法嵌入敌意，使得自动化系统难以准确解读。尽管当前的视觉-语言模型（VLMs）在显性案例上表现良好，但其部署受到高推理成本和面对微妙内容时持续失效的限制。本研究探讨了如何通过提示优化、微调和自动化数据增强来提升小型模型的性能。我们提出了一种端到端的流程，通过改变提示结构、标签粒度和训练模态，证明结构化提示和分级监督能显著增强紧凑型VLMs的性能。我们还开发了一个多模态增强框架，通过协调的LLM-VLM设置生成反事实中性模因，从而减少伪相关性并提升对隐晦仇恨内容的检测能力。消融实验量化了各组件的贡献，表明提示设计、细粒度标签和针对性增强共同缩小了小型与大型模型之间的差距。研究结果为构建更鲁棒、更易部署的多模态仇恨检测系统提供了一条实用路径，且无需依赖高成本的大模型推理。