Large language models (LLMs) have received increasing attention. However, due to the complexity of its capabilities, how to rationally evaluate the capabilities of LLMs is still a task to be solved. We propose the RoCar method, which utilizes the defined basic schemas to randomly construct a task graph and generates natural language evaluation tasks based on the task graph to evaluate the reasoning and memory abilities of LLMs respectively. Due to the very large randomness of the task construction process, it is possible to ensure that none of the LLMs to be tested has directly learned the evaluation tasks, guaranteeing the fairness of the evaluation method.

翻译：大语言模型（LLMs）日益受到关注。然而，由于其能力复杂性，如何合理评估LLMs的能力仍是一个有待解决的问题。我们提出RoCar方法，该方法利用定义的基本模式随机构建任务图，并基于该任务图生成自然语言评估任务，以分别评估LLMs的推理与记忆能力。由于任务构建过程具有极大的随机性，可确保任何被测LLMs均未直接学习过评估任务，从而保障了评估方法的公平性。