This paper introduces GraphOmni, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs on graph-theoretic tasks articulated in natural language. GraphOmni encompasses diverse graph types, serialization formats, and prompting schemes, significantly exceeding prior efforts in both scope and depth. Through extensive systematic evaluation, we identify critical interactions among these dimensions, demonstrating their substantial impact on model performance. Our experiments reveal that state-of-the-art models like Claude-3.5 and o4-mini consistently outperform other models, yet even these leading models exhibit substantial room for improvement. Performance variability is evident depending on the specific combinations of factors we considered, underscoring the necessity of comprehensive evaluations across these interconnected dimensions. Additionally, we observe distinct impacts of serialization and prompting strategies between open-source and closed-source models, encouraging the development of tailored approaches. Motivated by the findings, we also propose a reinforcement learning-inspired framework that adaptively selects the optimal factors influencing LLM reasoning capabilities. This flexible and extendable benchmark not only deepens our understanding of LLM performance on structured tasks but also provides a robust foundation for advancing research in LLM-based graph reasoning. The code and datasets are available at https://github.com/GAI-Community/GraphOmni.

翻译：本文介绍GraphOmni，这是一个为评估大语言模型在自然语言表述的图论任务上的推理能力而设计的综合性基准。GraphOmni涵盖了多样化的图类型、序列化格式和提示方案，在广度和深度上均显著超越了先前的工作。通过广泛的系统性评估，我们识别了这些维度之间的关键相互作用，证明了它们对模型性能的重大影响。我们的实验表明，Claude-3.5和o4-mini等最先进模型始终优于其他模型，但即使是这些领先模型也存在巨大的改进空间。性能变化明显取决于我们所考虑的具体因素组合，这强调了在这些相互关联的维度上进行全面评估的必要性。此外，我们观察到序列化和提示策略对开源模型与闭源模型产生了不同的影响，这鼓励了针对性方法的开发。基于这些发现，我们还提出了一个受强化学习启发的框架，该框架能自适应地选择影响大语言模型推理能力的最优因素。这个灵活且可扩展的基准不仅加深了我们对大语言模型在结构化任务上表现的理解，也为推进基于大语言模型的图推理研究提供了坚实的基础。代码和数据集可在 https://github.com/GAI-Community/GraphOmni 获取。