In many legal processes being able to action on the concrete implication of a legal question can be valuable to automating human review or signalling certain conditions (e.g., alerts around automatic renewal). To support such tasks, we present a form of legal question answering that seeks to return one (or more) fixed answers for a question about a contract clause. After showing that unstructured generative question answering can have questionable outcomes for such a task, we discuss our exploration methodology for legal question answering prompts using OpenAI's \textit{GPT-3.5-Turbo} and provide a summary of insights. Using insights gleaned from our qualitative experiences, we compare our proposed template prompts against a common semantic matching approach and find that our prompt templates are far more accurate despite being less reliable in the exact response return. With some additional tweaks to prompts and the use of in-context learning, we are able to further improve the performance of our proposed strategy while maximizing the reliability of responses as best we can.

翻译：在许多法律流程中，能够针对法律问题的具体含义采取行动，对于自动化人工审查或标记特定条件（如自动续约提醒）可能很有价值。为支持此类任务，我们提出了一种法律问题解答形式，旨在针对合同条款的问题返回一个（或多个）固定答案。在展示了非结构化生成式问答在此类任务中可能产生问题结果后，我们讨论了使用OpenAI的\textit{GPT-3.5-Turbo}进行法律问题解答提示的探索方法，并总结了关键见解。利用从定性经验中获得的洞见，我们将提出的模板提示与常见的语义匹配方法进行了比较，发现尽管模板提示在精确响应返回方面可靠性较低，但其准确性远高于后者。通过对提示进行额外调整并应用上下文学习，我们能够在尽可能最大化响应可靠性的同时，进一步提高所提出策略的性能。