The quest for human imitative AI has been an enduring topic in AI research since its inception. The technical evolution and emerging capabilities of the latest cohort of large language models (LLMs) have reinvigorated the subject beyond academia to the cultural zeitgeist. While recent NLP evaluation benchmark tasks test some aspects of human-imitative behaviour (e.g., BIG-bench's 'human-like behavior' tasks), few, if not none, examine creative problem solving abilities. Creative problem solving in humans is a well-studied topic in cognitive neuroscience with standardized tests that predominantly use the ability to associate (heterogeneous) connections among clue words as a metric for creativity. Exposure to misleading stimuli - distractors dubbed red herrings - impede human performance in such tasks via the fixation effect and Einstellung paradigm. In cognitive neuroscience studies, such fixations are experimentally induced by pre-exposing participants to orthographically similar incorrect words to subsequent word-fragments or clues. The popular British quiz show Only Connect's Connecting Wall segment essentially mimics Mednick's Remote Associates Test (RAT) formulation with built-in, deliberate red herrings, which makes it an ideal proxy dataset to explore and study fixation effect and Einstellung paradigm from cognitive neuroscience in LLMs. In this paper we present the novel Only Connect Wall (OCW) dataset and report results from our evaluation of selected pre-trained language models and LLMs on creative problem solving tasks like grouping clue words by heterogeneous connections, and identifying correct open knowledge domain connections in respective groups. We synthetically generate two additional datasets: OCW-Randomized, OCW-WordNet to further analyze our red-herrings hypothesis in language models. The code and link to the dataset are available at https://github.com/TaatiTeam/OCW.

翻译：自人工智能（AI）研究诞生以来，追求类人AI始终是其核心主题。最新一代大型语言模型（LLMs）的技术演进与新兴能力，使这一课题超越学术界并融入文化思潮。尽管近期自然语言处理（NLP）评估基准任务（如BIG-bench的“类人行为”任务）测试了人类模仿行为的某些方面，但鲜有（甚至没有）研究涉及创造性问题解决能力。在认知神经科学中，人类创造性问题解决是一个经过充分研究的课题，其标准化测试主要利用关联（异质）线索词的能力作为创造力指标。暴露于误导性刺激（即所谓的“红鲱鱼”干扰项）会通过固着效应与“定势范式”削弱人类在此类任务中的表现。在认知神经科学实验中，这类固着效应通过预先向参与者展示与后续词片段或线索存在正字法相似性的错误词语来诱发。英国热门益智节目《Only Connect》中的“连接墙”环节本质上模仿了梅德尼克的远程联想测试（RAT）范式，并内置了精心设计的红鲱鱼干扰项，使其成为探索LLMs中认知神经科学固着效应与定势范式的理想代理数据集。本文提出新型Only Connect Wall（OCW）数据集，并报告我们评估选定预训练语言模型与LLMs在创造性问题解决任务（如通过异质关联分组线索词、识别各组中正确的开放知识领域关联）上的结果。我们进一步合成生成两个额外数据集OCW-Randomized与OCW-WordNet，以深入分析语言模型中的“红鲱鱼假设”。代码与数据集链接见https://github.com/TaatiTeam/OCW。