This paper introduces GraphOmni, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs on graph-theoretic tasks articulated in natural language. GraphOmni encompasses diverse graph types, serialization formats, and prompting schemes, significantly exceeding prior efforts in both scope and depth. Through extensive systematic evaluation, we identify critical interactions among these dimensions, demonstrating their substantial impact on model performance. Our experiments reveal that state-of-the-art models like Claude-3.5 and o4-mini consistently outperform other models, yet even these leading models exhibit substantial room for improvement. Performance variability is evident depending on the specific combinations of factors we considered, underscoring the necessity of comprehensive evaluations across these interconnected dimensions. Additionally, we observe distinct impacts of serialization and prompting strategies between open-source and closed-source models, encouraging the development of tailored approaches. Motivated by the findings, we also propose a reinforcement learning-inspired framework that adaptively selects the optimal factors influencing LLM reasoning capabilities. This flexible and extendable benchmark not only deepens our understanding of LLM performance on structured tasks but also provides a robust foundation for advancing research in LLM-based graph reasoning. The code and datasets are available at https://github.com/GAI-Community/GraphOmni.

翻译：本文介绍GraphOmni，一个综合基准框架，旨在评估大语言模型在自然语言描述的图理论任务上的推理能力。GraphOmni涵盖多种图类型、序列化格式和提示方案，在广度和深度上显著超越此前研究。通过系统的广泛评估，我们发现了这些维度间的关键交互作用，证明了它们对模型性能的重大影响。实验表明，Claude-3.5和o4-mini等最先进模型始终优于其他模型，但这些领先模型仍存在显著改进空间。根据我们考虑的特定因素组合，模型性能存在明显差异，这凸显了在这些相互关联的维度上进行全面评估的必要性。此外，我们观察到开源模型与闭源模型在序列化和提示策略上的不同影响，这鼓励了针对性方法的发展。基于这些发现，我们还提出了一种受强化学习启发的框架，可自适应选择影响LLM推理能力的最优因素。这一灵活且可扩展的基准不仅加深了我们对LLM在结构化任务上表现的理解，也为推进基于LLM的图推理研究提供了坚实基础。代码和数据集见https://github.com/GAI-Community/GraphOmni。