Large language models (LLMs) are able to do accurate classification with zero or only a few examples (in-context learning). We show a prompting system that enables regression with uncertainty for in-context learning with frozen LLM (GPT-3, GPT-3.5, and GPT-4) models, allowing predictions without features or architecture tuning. By incorporating uncertainty, our approach enables Bayesian optimization for catalyst or molecule optimization using natural language, eliminating the need for training or simulation. Here, we performed the optimization using the synthesis procedure of catalysts to predict properties. Working with natural language mitigates difficulty synthesizability since the literal synthesis procedure is the model's input. We showed that in-context learning could improve past a model context window (maximum number of tokens the model can process at once) as data is gathered via example selection, allowing the model to scale better. Although our method does not outperform all baselines, it requires zero training, feature selection, and minimal computing while maintaining satisfactory performance. We also find Gaussian Process Regression on text embeddings is strong at Bayesian optimization. The code is available in our GitHub repository: https://github.com/ur-whitelab/BO-LIFT

翻译：大型语言模型（LLMs）能够在零样本或仅需少量样本（上下文学习）的情况下进行精准分类。我们提出了一种提示系统，使得冻结的LLM模型（GPT-3、GPT-3.5和GPT-4）能够通过上下文学习实现带不确定性的回归，从而无需特征提取或架构调优即可进行预测。通过引入不确定性，我们的方法能够利用自然语言实现对催化剂或分子的贝叶斯优化，消除了训练或仿真的需求。本研究基于催化剂的合成流程进行属性预测优化。由于模型直接以文字描述的合成流程作为输入，使用自然语言有效降低了合成可行性评估的难度。我们证明，通过示例筛选积累数据，上下文学习能够突破模型上下文窗口（模型单次处理的最大令牌数）的限制，从而提升模型的可扩展性。尽管我们的方法未能超越所有基线模型，但它无需训练、特征选择且计算需求极低，同时保持了令人满意的性能。此外，我们发现基于文本嵌入的高斯过程回归在贝叶斯优化中表现优异。相关代码已发布于GitHub仓库：https://github.com/ur-whitelab/BO-LIFT