The quest for human imitative AI has been an enduring topic in AI research since its inception. The technical evolution and emerging capabilities of the latest cohort of large language models (LLMs) have reinvigorated the subject beyond academia to the cultural zeitgeist. While recent NLP evaluation benchmark tasks test some aspects of human-imitative behaviour (e.g., BIG-bench's 'human-like behavior' tasks), few, if not none, examine creative problem solving abilities. Creative problem solving in humans is a well-studied topic in cognitive neuroscience with standardized tests that predominantly use the ability to associate (heterogeneous) connections among clue words as a metric for creativity. Exposure to misleading stimuli - distractors dubbed red herrings - impede human performance in such tasks via the fixation effect and Einstellung paradigm. In cognitive neuroscience studies, such fixations are experimentally induced by pre-exposing participants to orthographically similar incorrect words to subsequent word-fragments or clues. The popular British quiz show Only Connect's Connecting Wall segment essentially mimics Mednick's Remote Associates Test (RAT) formulation with built-in, deliberate red herrings, which makes it an ideal proxy dataset to explore and study fixation effect and Einstellung paradigm from cognitive neuroscience in LLMs. In addition to presenting the novel Only Connect Wall (OCW) dataset, we also report results from our evaluation of selected pre-trained language models and LLMs (including OpenAI's GPT series) on creative problem solving tasks like grouping clue words by heterogeneous connections, and identifying correct open knowledge domain connections in respective groups. The code and link to the dataset are available at https://github.com/TaatiTeam/OCW.

翻译：自人工智能研究诞生以来，实现类人智能模拟始终是该领域的持久课题。最新一代大语言模型的技术演进与新兴能力将这一主题从学术界重新推向文化潮流前沿。尽管近期自然语言处理评估基准任务（如BIG-bench的"类人行为"任务）测试了部分人类模仿行为特征，但鲜有研究深入考察创造性问题求解能力。在认知神经科学中，人类创造性问题求解是经过充分研究的课题，其标准化测试主要采用线索词（异构）关联能力作为创造力度量指标。接触误导性刺激（即被称为"红鲱鱼"的干扰项）会通过思维固着效应与定势范式削弱人类在此类任务中的表现。认知神经科学研究通过让受试者预先接触与后续词片或线索在拼写上相近的错误词汇，实验性地诱导此类思维固着。英国知名益智节目《Only Connect》的"连接墙"环节本质上模仿了梅德尼克的远程联想测验范式，并内置了精心设计的干扰项，这使其成为探索大语言模型中认知神经科学思维固着效应与定势范式的理想代理数据集。除构建全新的Only Connect Wall数据集外，我们还报告了选定预训练语言模型与大型语言模型（包括OpenAI的GPT系列）在创造性问题求解任务上的评估结果，涵盖基于异构关联的线索词分组及识别各组正确开放知识域连接等任务。数据集代码及链接详见https://github.com/TaatiTeam/OCW。