Large Language Models (LLMs), such as ChatGPT, have drawn a lot of attentions recently in the legal domain due to its emergent ability to tackle a variety of legal tasks. However, it is still unknown if LLMs are able to analyze a legal case and perform reasoning in the same manner as lawyers. Therefore, we constructed a novel corpus consisting of scenarios pertain to Contract Acts Malaysia and Australian Social Act for Dependent Child. ChatGPT is applied to perform analysis on the corpus using the IRAC method, which is a framework widely used by legal professionals for organizing legal analysis. Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations. In addition, we conducted the first empirical assessment of ChatGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals. Our experimental results shed lights on possible future research directions to improve alignments between LLMs and legal experts in terms of legal reasoning.

翻译：大型语言模型（LLMs），如ChatGPT，近年来因其在多种法律任务中展现出的涌现能力而备受法律领域关注。然而，LLMs能否像律师一样分析法律案例并进行推理仍属未知。为此，我们构建了一个包含《马来西亚合同法》和《澳大利亚受抚养子女社会法案》相关场景的新型语料库。我们采用法律专业人士广泛用于组织法律分析的IRAC框架，对ChatGPT进行语料分析测试。该语料库中的每个场景均以半结构化格式标注了完整的IRAC分析，使机器与法律专业人士均能解读和理解标注内容。此外，我们首次对ChatGPT进行IRAC分析能力的实证评估，旨在探究其与法律专业人士分析结果的一致性程度。实验结果为改进LLMs与法律专家在法律推理层面的对齐性指明了未来可能的研究方向。