Large Language Models (LLMs), such as ChatGPT, have drawn a lot of attentions recently in the legal domain due to its emergent ability to tackle a variety of legal tasks. However, it is still unknown if LLMs are able to analyze a legal case and perform reasoning in the same manner as lawyers. Therefore, we constructed a novel corpus consisting of scenarios pertain to Contract Acts Malaysia and Australian Social Act for Dependent Child. ChatGPT is applied to perform analysis on the corpus using the IRAC method, which is a framework widely used by legal professionals for organizing legal analysis. Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format so that both machines and legal professionals are able to interpret and understand the annotations. In addition, we conducted the first empirical assessment of ChatGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals. Our experimental results shed lights on possible future research directions to improve alignments between LLMs and legal experts in terms of legal reasoning.

翻译：大型语言模型（LLMs），如ChatGPT，近期因其在多种法律任务中展现的涌现能力而引发法律领域的广泛关注。然而，LLMs能否像律师一样分析法律案例并进行推理仍属未知。为此，我们构建了一个包含马来西亚合同法及澳大利亚受抚养子女社会法案相关场景的新颖语料库。采用IRAC方法——法律专业人士广泛用于组织法律分析的框架——使ChatGPT对该语料库进行分析。语料库中的每个场景均以半结构化格式标注完整的IRAC分析，以便机器与法律专业人士均能解读和理解标注内容。此外，我们首次对ChatGPT进行IRAC分析实证评估，以探究其与法律专业人士分析的契合程度。实验结果揭示了未来改进LLMs与法律专家在法律推理方面对齐性的潜在研究方向。