As Large Language Models (LLMs) are proposed as legal decision assistants, and even first-instance decision-makers, across a range of judicial and administrative contexts, it becomes essential to explore how they answer legal questions, and in particular the factors that lead them to decide difficult questions in one way or another. A specific feature of legal decisions is the need to respond to arguments advanced by contending parties. A legal decision-maker must be able to engage with, and respond to, including through being potentially persuaded by, arguments advanced by the parties. Conversely, they should not be unduly persuadable, influenced by a particularly compelling advocate to decide cases based on the skills of the advocates, rather than the merits of the case. We explore how frontier open- and closed-weights LLMs respond to legal arguments, reporting original experimental results examining how the quality of the advocate making those arguments affects the likelihood that a model will agree with a particular legal point of view, and exploring the factors driving these results. Our results have implications for the feasibility of adopting LLMs across legal and administrative settings.

翻译：随着大型语言模型（LLMs）被提议作为一系列司法和行政场景中的法律决策辅助工具，甚至作为初审决策者，探索它们如何回答法律问题——特别是哪些因素促使它们以某种方式裁决疑难问题——变得至关重要。法律决策的一个特定特征是需要回应争议双方提出的论点。法律决策者必须能够理解并回应——包括可能被当事人的论点所说服。反之，决策者不应过度易被说服，不应受特别有说服力的辩护人影响，导致基于辩护人的技巧而非案件是非曲直作出裁决。我们探究前沿开源和闭源权重大型语言模型如何回应法律论点，报告原创实验结果，考察辩护人提出论点的质量如何影响模型赞同特定法律观点的可能性，并探讨驱动这些结果的因素。我们的研究结果对在法律及行政环境中采用大型语言模型的可行性具有启示意义。