Supramolecular chemistry, which includes the study of non-covalent host-guest assemblies, has advanced various applications. However, designing host-guest systems remains time-consuming, requiring days of dry-lab verification per candidate pair. Although LLMs have emerged as a fast alternative with strong performance on molecular binding tasks, no benchmark currently systematically evaluates LLMs for host-guest reasoning across fundamental supramolecular chemistry tasks, e.g., binding affinity prediction. To this end, we collaborate with domain experts to release the first Supramolecular Benchmark, called SupraBench, to evaluate LLMs in chemistry reasoning. Specifically, we design four fundamental tasks, i.e., binding affinity prediction, top-binder selection, solvent identification, and host-guest description, plus an auxiliary vision-based task for molecular identification. We also release SupraPMC, a curated 16M-token corpus of Supramolecular chemistry articles distilled from Europe PMC, to support the adaptation to the supramolecular domain. We benchmark a broad range of open and proprietary LLMs and find that LLMs leave substantial headroom across all tasks. Domain adaptation pretraining over SupraPMC transfers cleanly to in-distribution regression but trades off against strict letter-format output. Moreover, the difficulty profile differs sharply across task families, revealing distinct failure modes that indicate specific gaps in current supramolecular chemistry reasoning. Our source codes and benchmark datasets are available at https://github.com/Tianyi-Billy-Ma/SupraBench.

翻译：超分子化学涵盖非共价主客体组装的研究，推动了多种应用的发展。然而，设计主客体系统仍耗时费力，每个候选对需经过数天的干实验验证。尽管大语言模型（LLMs）作为快速替代方案在分子结合任务中展现出强劲性能，但目前尚缺乏系统评估LLMs在超分子化学基础任务（如结合亲和力预测）中进行主客体推理的基准测试。为此，我们与领域专家合作发布首个超分子化学基准测试——SupraBench，用于评估LLMs在化学推理中的表现。具体而言，我们设计了四项基础任务（结合亲和力预测、顶级结合剂选择、溶剂识别、主客体描述）以及一项辅助性的基于视觉的分子识别任务。同时，我们发布了SupraPMC——一个从Europe PMC提取并精选的包含1600万词元的超分子化学文章语料库，支持超分子领域的适配。我们对多种开源与专有LLMs进行了基准测试，发现LLMs在所有任务上均存在显著提升空间。基于SupraPMC的领域自适应预训练能有效迁移至分布内回归任务，但会与严格的字母格式输出产生权衡。此外，不同任务家族的难度分布差异显著，揭示了LLMs在超分子化学推理中的特定失败模式与能力缺口。我们的源代码与基准数据集已开源至https://github.com/Tianyi-Billy-Ma/SupraBench。