Classical Chinese Understanding (CCU) holds significant value in preserving and exploration of the outstanding traditional Chinese culture. Recently, researchers have attempted to leverage the potential of Large Language Models (LLMs) for CCU by capitalizing on their remarkable comprehension and semantic capabilities. However, no comprehensive benchmark is available to assess the CCU capabilities of LLMs. To fill this gap, this paper introduces C$^{3}$bench, a Comprehensive Classical Chinese understanding benchmark, which comprises 50,000 text pairs for five primary CCU tasks, including classification, retrieval, named entity recognition, punctuation, and translation. Furthermore, the data in C$^{3}$bench originates from ten different domains, covering most of the categories in classical Chinese. Leveraging the proposed C$^{3}$bench, we extensively evaluate the quantitative performance of 15 representative LLMs on all five CCU tasks. Our results not only establish a public leaderboard of LLMs' CCU capabilities but also gain some findings. Specifically, existing LLMs are struggle with CCU tasks and still inferior to supervised models. Additionally, the results indicate that CCU is a task that requires special attention. We believe this study could provide a standard benchmark, comprehensive baselines, and valuable insights for the future advancement of LLM-based CCU research. The evaluation pipeline and dataset are available at \url{https://github.com/SCUT-DLVCLab/C3bench}.

翻译：古文理解对于传承与发掘中华优秀传统文化具有重要价值。近期，研究者尝试利用大语言模型卓越的理解与语义能力，探索其在古文理解任务中的潜力。然而，目前尚缺乏一个全面的基准来系统评估大语言模型的古文理解能力。为填补这一空白，本文提出了C$^{3}$bench——一个综合性的古文理解基准，该基准包含50,000个文本对，涵盖分类、检索、命名实体识别、断句与翻译五大核心古文理解任务。此外，C$^{3}$bench的数据来源于十个不同领域，覆盖了古文的大部分类别。基于所提出的C$^{3}$bench，我们对15个代表性大语言模型在所有五项古文理解任务上的量化性能进行了广泛评估。我们的实验结果不仅建立了一个公开的大语言模型古文理解能力排行榜，还获得了一些重要发现。具体而言，现有大语言模型在古文理解任务上仍面临困难，其表现仍逊于有监督模型。此外，结果表明古文理解是一项需要特别关注的任务。我们相信这项研究能为未来基于大语言模型的古文理解研究提供一个标准基准、全面的基线以及有价值的见解。评估流程与数据集已公开于 \url{https://github.com/SCUT-DLVCLab/C3bench}。