Large Language Models (LLMs) stand at the forefront of a number of Natural Language Processing (NLP) tasks. Despite the widespread adoption of LLMs in NLP, much of their potential in broader fields remains largely unexplored, and significant limitations persist in their design and implementation. Notably, LLMs struggle with structured data, such as graphs, and often falter when tasked with answering domain-specific questions requiring deep expertise, such as those in biology and chemistry. In this paper, we explore a fundamental question: Can LLMs effectively handle molecule prediction tasks? Rather than pursuing top-tier performance, our goal is to assess how LLMs can contribute to diverse molecule tasks. We identify several classification and regression prediction tasks across six standard molecule datasets. Subsequently, we carefully design a set of prompts to query LLMs on these tasks and compare their performance with existing Machine Learning (ML) models, which include text-based models and those specifically designed for analysing the geometric structure of molecules. Our investigation reveals several key insights: Firstly, LLMs generally lag behind ML models in achieving competitive performance on molecule tasks, particularly when compared to models adept at capturing the geometric structure of molecules, highlighting the constrained ability of LLMs to comprehend graph data. Secondly, LLMs show promise in enhancing the performance of ML models when used collaboratively. Lastly, we engage in a discourse regarding the challenges and promising avenues to harness LLMs for molecule prediction tasks. The code and models are available at https://github.com/zhiqiangzhongddu/LLMaMol.

翻译：【翻译摘要】 大型语言模型在众多自然语言处理任务中处于前沿地位。尽管LLMs在NLP领域已被广泛采用，但其在更广泛领域的潜力仍大多未被探索，且在其设计与实现中存在显著局限性。值得注意的是，LLMs在处理结构化数据（如图结构）时面临困难，且在需要深度领域知识（如生物学与化学）的特定领域问题回答中时常表现不佳。本文探究一个基础性问题：LLMs能否有效处理分子预测任务？我们的目标并非追求顶尖性能，而是评估LLMs如何为多样化分子任务做出贡献。我们基于六个标准分子数据集确定了多项分类与回归预测任务，继而精心设计一组提示词以查询LLMs在这些任务上的表现，并将其与现有机器学习模型（包括基于文本的模型及专门分析分子几何结构的模型）进行性能对比。研究揭示了若干关键发现：首先，在分子任务上，LLMs整体上落后于ML模型——尤其相较于擅长捕捉分子几何结构的模型，凸显了LLMs理解图数据能力的局限。其次，当与ML模型协同使用时，LLMs展现出提升其性能的潜力。最后，我们围绕利用LLMs完成分子预测任务所面临的挑战与可行方向展开讨论。相关代码与模型已开源至https://github.com/zhiqiangzhongddu/LLMaMol。