The quest for human imitative AI has been an enduring topic in AI research since its inception. The technical evolution and emerging capabilities of the latest cohort of large language models (LLMs) have reinvigorated the subject beyond academia to the cultural zeitgeist. While recent NLP evaluation benchmark tasks test some aspects of human-imitative behaviour (e.g., BIG-bench's 'human-like behavior' tasks), few, if not none, examine creative problem solving abilities. Creative problem solving in humans is a well-studied topic in cognitive neuroscience with standardized tests that predominantly use the ability to associate (heterogeneous) connections among clue words as a metric for creativity. Exposure to misleading stimuli - distractors dubbed red herrings - impede human performance in such tasks via the fixation effect and Einstellung paradigm. In cognitive neuroscience studies, such fixations are experimentally induced by pre-exposing participants to orthographically similar incorrect words to subsequent word-fragments or clues. The popular British quiz show Only Connect's Connecting Wall segment essentially mimics Mednick's Remote Associates Test (RAT) formulation with built-in, deliberate red herrings, which makes it an ideal proxy dataset to explore and study fixation effect and Einstellung paradigm from cognitive neuroscience in LLMs. In this paper we present the novel Only Connect Wall (OCW) dataset and report results from our evaluation of selected pre-trained language models and LLMs on creative problem solving tasks like grouping clue words by heterogeneous connections, and identifying correct open knowledge domain connections in respective groups. We synthetically generate two additional datasets: OCW-Randomized, OCW-WordNet to further analyze our red-herrings hypothesis in language models. The code and link to the dataset are available at https://github.com/TaatiTeam/OCW.

翻译：自人工智能研究诞生以来，追求类人智能一直是其持久主题。最新一代大型语言模型（LLM）的技术演进与新兴能力，使这一课题超越学术范畴，成为文化思潮的焦点。尽管近期自然语言处理（NLP）评估基准任务测试了某些类人行为（如BIG-bench的“类人行为”任务），但鲜有研究涉及创造性问题解决能力。人类创造性问题解决是认知神经科学中研究充分的课题，其标准化测试主要依据关联线索词（异质连接）的能力作为创造力指标。暴露于误导性刺激（即所谓的“红鲱鱼”干扰项）会通过固着效应和定势范式阻碍人类任务表现。在认知神经科学研究中，此类固着通过让参与者预先接触与后续词素或线索正字法相似的错误词语实验诱导。英国热门智力竞赛节目《Only Connect》的“谜墙”环节本质上模仿了梅德尼克的远距离联想测验（RAT）构型，并内置刻意设计的“红鲱鱼”，使其成为探索LLM中认知神经科学固着效应与定势范式的理想代理数据集。本文提出新颖的Only Connect Wall（OCW）数据集，并报告所选预训练语言模型与LLM在创造性问题解决任务（如按异质连接对线索词分组、识别各组内正确的开放知识领域连接）上的评估结果。我们进一步合成生成OCW-Randomized和OCW-WordNet两个附加数据集，以深入分析语言模型中的“红鲱鱼”假设。数据集代码及链接见https://github.com/TaatiTeam/OCW。