In this paper, we address the challenge of detecting hateful memes in the low-resource setting where only a few labeled examples are available. Our approach leverages the compositionality of Low-rank adaptation (LoRA), a widely used parameter-efficient tuning technique. We commence by fine-tuning large language models (LLMs) with LoRA on selected tasks pertinent to hateful meme detection, thereby generating a suite of LoRA modules. These modules are capable of essential reasoning skills for hateful meme detection. We then use the few available annotated samples to train a module composer, which assigns weights to the LoRA modules based on their relevance. The model's learnable parameters are directly proportional to the number of LoRA modules. This modularized network, underpinned by LLMs and augmented with LoRA modules, exhibits enhanced generalization in the context of hateful meme detection. Our evaluation spans three datasets designed for hateful meme detection in a few-shot learning context. The proposed method demonstrates superior performance to traditional in-context learning, which is also more computationally intensive during inference.We then use the few available annotated samples to train a module composer, which assigns weights to the LoRA modules based on their relevance. The model's learnable parameters are directly proportional to the number of LoRA modules. This modularized network, underpinned by LLMs and augmented with LoRA modules, exhibits enhanced generalization in the context of hateful meme detection. Our evaluation spans three datasets designed for hateful meme detection in a few-shot learning context. The proposed method demonstrates superior performance to traditional in-context learning, which is also more computationally intensive during inference.

翻译：本文针对低资源场景下仅拥有少量标注样本的仇恨表情包检测挑战，提出一种基于低秩适应（LoRA）参数高效微调技术组合性的方法。我们首先在仇恨表情包检测相关选定的任务上，使用LoRA对大型语言模型（LLMs）进行微调，生成一系列LoRA模块。这些模块具备仇恨表情包检测所需的核心推理能力。随后，我们利用少量可用标注样本训练一个模块组合器，该组合器根据相关性为LoRA模块分配权重。模型的可学习参数数量与LoRA模块数量成正比。这种以LLMs为基础、以LoRA模块增强的模块化网络，在仇恨表情包检测场景中展现出更强的泛化能力。我们在三个专为少样本学习背景下仇恨表情包检测设计的数据集上进行了评估。实验表明，所提方法优于传统上下文学习——后者在推理阶段计算密集度更高。