Hateful meme detection is a challenging multimodal task that requires comprehension of both vision and language, as well as cross-modal interactions. Recent studies have tried to fine-tune pre-trained vision-language models (PVLMs) for this task. However, with increasing model sizes, it becomes important to leverage powerful PVLMs more efficiently, rather than simply fine-tuning them. Recently, researchers have attempted to convert meme images into textual captions and prompt language models for predictions. This approach has shown good performance but suffers from non-informative image captions. Considering the two factors mentioned above, we propose a probing-based captioning approach to leverage PVLMs in a zero-shot visual question answering (VQA) manner. Specifically, we prompt a frozen PVLM by asking hateful content-related questions and use the answers as image captions (which we call Pro-Cap), so that the captions contain information critical for hateful content detection. The good performance of models with Pro-Cap on three benchmarks validates the effectiveness and generalization of the proposed method.

翻译：仇恨模因检测是一项挑战性的多模态任务，需要同时理解视觉与语言信息，并处理跨模态交互。近年研究尝试微调预训练视觉语言模型（PVLM）以完成该任务。然而，随着模型规模不断增大，更高效地利用强大的PVLM而非简单微调变得至关重要。近期，研究者尝试将模因图像转换为文本描述，并借助语言模型进行预测。该方法虽表现良好，但存在图像描述信息量不足的问题。综合上述两方面因素，我们提出一种基于探测的生成描述方法，以零样本视觉问答（VQA）的方式利用PVLM。具体而言，我们通过询问与仇恨内容相关的问题来提示冻结的PVLM，并将其回答作为图像描述（称为Pro-Cap），从而使描述包含检测仇恨内容的关键信息。在三个基准测试中，采用Pro-Cap的模型表现出色，验证了该方法的有效性和泛化能力。