Understanding the extent to which the perceptual world can be recovered from language is a fundamental problem in cognitive science. We reformulate this problem as that of distilling psychophysical information from text and show how this can be done by combining large language models (LLMs) with a classic psychophysical method based on similarity judgments. Specifically, we use the prompt auto-completion functionality of GPT3, a state-of-the-art LLM, to elicit similarity scores between stimuli and then apply multidimensional scaling to uncover their underlying psychological space. We test our approach on six perceptual domains and show that the elicited judgments strongly correlate with human data and successfully recover well-known psychophysical structures such as the color wheel and pitch spiral. We also explore meaningful divergences between LLM and human representations. Our work showcases how combining state-of-the-art machine models with well-known cognitive paradigms can shed new light on fundamental questions in perception and language research.

翻译：理解从语言中可恢复的感知世界程度是认知科学中的一个基本问题。我们将此问题重新定义为从文本中提取心理物理信息，并展示如何通过将大语言模型与基于相似性判断的经典心理物理方法相结合来实现这一目标。具体而言，我们利用最先进的大语言模型GPT3的提示自动补全功能，获取刺激之间的相似性评分，然后应用多维尺度分析揭示其潜在的心理空间。我们在六个感知领域测试了该方法，结果表明所获取的判断与人类数据高度相关，并成功恢复了众所周知的物理心理结构，如色轮和音高螺旋。我们还探讨了大语言模型与人类表征之间的有意义差异。我们的工作展示了将最先进的机器模型与经典认知范式相结合如何为感知和语言研究中的基本问题提供新视角。