Metaphor and humor share a lot of common ground, and metaphor is one of the most common humorous mechanisms. This study focuses on the humorous capacity of multimodal metaphors, which has not received due attention in the community. We take inspiration from the Incongruity Theory of humor, the Conceptual Metaphor Theory, and the annotation scheme behind the VU Amsterdam Metaphor Corpus, and developed a novel annotation scheme for humorous multimodal metaphor use in image-caption pairs. We create the Hummus Dataset of Humorous Multimodal Metaphor Use, providing expert annotation on 1k image-caption pairs sampled from the New Yorker Caption Contest corpus. Using the dataset, we test state-of-the-art multimodal large language models (MLLMs) on their ability to detect and understand humorous multimodal metaphor use. Our experiments show that current MLLMs still struggle with processing humorous multimodal metaphors, particularly with regard to integrating visual and textual information. We release our dataset and code at github.com/xiaoyuisrain/humorous-multimodal-metaphor-use.

翻译：隐喻与幽默具有诸多共同基础，隐喻是最常见的幽默机制之一。本研究聚焦于多模态隐喻的幽默生成能力，该领域尚未在学界获得足够重视。我们受幽默的不协调理论、概念隐喻理论及阿姆斯特丹自由大学隐喻语料库标注体系的启发，开发了一套针对图像-标题对中幽默多模态隐喻使用的新型标注方案。基于《纽约客》标题竞赛语料库中采样的1000个图像-标题对，我们构建了幽默多模态隐喻使用数据集（Hummus Dataset），并提供专家级标注。利用该数据集，我们测试了当前最先进的多模态大语言模型在检测和理解幽默多模态隐喻使用方面的能力。实验表明，现有多模态大语言模型在处理幽默多模态隐喻时仍面临挑战，尤其在整合视觉与文本信息方面存在不足。我们已将数据集与代码发布于github.com/xiaoyuisrain/humorous-multimodal-metaphor-use。