Weird, unusual, and uncanny images pique the curiosity of observers because they challenge commonsense. For example, an image released during the 2022 world cup depicts the famous soccer stars Lionel Messi and Cristiano Ronaldo playing chess, which playfully violates our expectation that their competition should occur on the football field. Humans can easily recognize and interpret these unconventional images, but can AI models do the same? We introduce WHOOPS!, a new dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. We consider several tasks posed over the dataset. In addition to image captioning, cross-modal matching, and visual question answering, we introduce a difficult explanation generation task, where models must identify and explain why a given image is unusual. Our results show that state-of-the-art models such as GPT3 and BLIP2 still lag behind human performance on WHOOPS!. We hope our dataset will inspire the development of AI models with stronger visual commonsense reasoning abilities. Data, models and code are available at the project website: whoops-benchmark.github.io

翻译：怪异、不寻常、令人费解的图像因挑战常识而激发观察者的好奇心。例如，2022年世界杯期间流传的一张图片描绘了著名球星莱昂内尔·梅西与克里斯蒂亚诺·罗纳尔多在下国际象棋，这戏谑地违背了我们对他们应在足球场上一决高下的预期。人类能轻松识别并解读这些非常规图像，但AI模型能否做到同样的事情？我们提出了WHOOPS！——一个用于视觉常识的新数据集与基准。该数据集由设计师使用公开可用的图像生成工具（如Midjourney）特意创作的违反常识的图像组成。我们针对数据集设想了多项任务。除图像描述、跨模态匹配与视觉问答外，我们引入了一项困难的解释生成任务，要求模型识别并解释给定图像的异常之处。结果表明，GPT3和BLIP2等最先进模型在WHOOPS！上的表现仍落后于人类。我们希望该数据集能激发具有更强视觉常识推理能力的AI模型的开发。数据、模型与代码可在项目网站whoops-benchmark.github.io获取。