Over decades, neuroscience has accumulated a wealth of research results in the text modality that can be used to explore cognitive processes. Meta-analysis is a typical method that successfully establishes a link from text queries to brain activation maps using these research results, but it still relies on an ideal query environment. In practical applications, text queries used for meta-analyses may encounter issues such as semantic redundancy and ambiguity, resulting in an inaccurate mapping to brain images. On the other hand, large language models (LLMs) like ChatGPT have shown great potential in tasks such as context understanding and reasoning, displaying a high degree of consistency with human natural language. Hence, LLMs could improve the connection between text modality and neuroscience, resolving existing challenges of meta-analyses. In this study, we propose a method called Chat2Brain that combines LLMs to basic text-2-image model, known as Text2Brain, to map open-ended semantic queries to brain activation maps in data-scarce and complex query environments. By utilizing the understanding and reasoning capabilities of LLMs, the performance of the mapping model is optimized by transferring text queries to semantic queries. We demonstrate that Chat2Brain can synthesize anatomically plausible neural activation patterns for more complex tasks of text queries.

翻译：数十年来，神经科学在文本模态中积累了丰富的研究成果，可用于探索认知过程。元分析是一种典型方法，能够利用这些研究成果将文本查询成功关联到脑激活图，但其仍依赖于理想的查询环境。在实际应用中，用于元分析的文本查询可能面临语义冗余与歧义等问题，导致其与脑图像的映射不准确。另一方面，像ChatGPT这样的大型语言模型（LLMs）在上下文理解与推理等任务中展现出巨大潜力，与人类自然语言具有高度一致性。因此，LLMs有望改善文本模态与神经科学之间的连接，解决元分析现存的挑战。本研究提出一种名为Chat2Brain的方法，将LLMs与基础文本到图像模型（即Text2Brain）相结合，在数据稀缺及复杂查询环境下将开放式语义查询映射到脑激活图。通过利用LLMs的理解与推理能力，将文本查询转换为语义查询，从而优化映射模型的性能。我们证明，Chat2Brain能够针对更复杂的文本查询任务，合理解析出解剖学上可信的神经激活模式。