Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain. However, recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. To systematically investigate their competency in the law, we design practical baseline solutions based on LLMs and test on the task of legal judgment prediction. In our solutions, LLMs can work alone to answer open questions or coordinate with an information retrieval (IR) system to learn from similar cases or solve simplified multi-choice questions. We show that similar cases and multi-choice options, namely label candidates, included in prompts can help LLMs recall domain knowledge that is critical for expertise legal reasoning. We additionally present an intriguing paradox wherein an IR system surpasses the performance of LLM+IR due to limited gains acquired by weaker LLMs from powerful IR systems. In such cases, the role of LLMs becomes redundant. Our evaluation pipeline can be easily extended into other tasks to facilitate evaluations in other domains. Code is available at https://github.com/srhthu/LM-CompEval-Legal

翻译：大型语言模型（LLMs）在法律等专业领域展现出巨大潜力。然而，近期关于GPT-4法律评估的争议引发了对其在实际法律任务中表现的质疑。为系统考察LLMs在法律领域的胜任能力，我们设计了基于LLMs的实用基线解决方案，并在法律判决预测任务上进行了测试。在我们的方案中，LLMs可单独运行以回答开放式问题，或与信息检索（IR）系统协同工作，从相似案例中学习或解决简化后的多项选择题。研究表明，提示中包含的相似案例与多选题选项（即标签候选）能帮助LLMs唤醒专业知识推理所需的关键领域知识。此外，我们揭示了一个有趣悖论：当弱LLMs从强IR系统中获得的增益有限时，单一IR系统的表现甚至优于LLM+IR组合。在这种情况下，LLMs的作用变得冗余。我们的评估流程可便捷扩展至其他任务，促进跨领域评估。代码见https://github.com/srhthu/LM-CompEval-Legal