Large language models (LLMs) have made significant progress in natural language processing tasks and demonstrate considerable potential in the legal domain. However, legal applications demand high standards of accuracy, reliability, and fairness. Applying existing LLMs to legal systems without careful evaluation of their potential and limitations could pose significant risks in legal practice. To this end, we introduce a standardized comprehensive Chinese legal benchmark LexEval. This benchmark is notable in the following three aspects: (1) Ability Modeling: We propose a new taxonomy of legal cognitive abilities to organize different tasks. (2) Scale: To our knowledge, LexEval is currently the largest Chinese legal evaluation dataset, comprising 23 tasks and 14,150 questions. (3) Data: we utilize formatted existing datasets, exam datasets and newly annotated datasets by legal experts to comprehensively evaluate the various capabilities of LLMs. LexEval not only focuses on the ability of LLMs to apply fundamental legal knowledge but also dedicates efforts to examining the ethical issues involved in their application. We evaluated 38 open-source and commercial LLMs and obtained some interesting findings. The experiments and findings offer valuable insights into the challenges and potential solutions for developing Chinese legal systems and LLM evaluation pipelines. The LexEval dataset and leaderboard are publicly available at \url{https://github.com/CSHaitao/LexEval} and will be continuously updated.

翻译：大语言模型（LLMs）在自然语言处理任务中取得了显著进展，并在法律领域展现出巨大潜力。然而，法律应用对准确性、可靠性和公平性有着极高要求。若未经审慎评估其潜力与局限便将现有大语言模型应用于法律系统，可能给司法实践带来重大风险。为此，我们提出了标准化的中文综合性法律基准LexEval。该基准在以下三个方面具有突出特点：（1）能力建模：我们提出了一种新的法律认知能力分类体系来组织不同任务。（2）规模：据我们所知，LexEval是目前规模最大的中文法律评估数据集，包含23项任务共计14,150道题目。（3）数据构成：我们整合了结构化现有数据集、考试数据集及经法律专家新标注的数据集，以全面评估大语言模型的各项能力。LexEval不仅关注大语言模型应用基础法律知识的能力，还着力考察其应用过程中涉及的伦理问题。我们评估了38个开源与商业大语言模型，并获得若干重要发现。实验结果为开发中文法律系统与大语言模型评估流程所面临的挑战及潜在解决方案提供了宝贵洞见。LexEval数据集与排行榜已公开于\url{https://github.com/CSHaitao/LexEval}，并将持续更新。