Reasoning benchmarks have played a crucial role in the progress of language models. Yet rigorous evaluation remains a significant challenge as static question-answer pairs provide only a snapshot of performance, compressing complex behavior into a single accuracy metric. This limitation is especially true in complex, rule-bound domains such as law, where existing benchmarks are costly to build and ill suited for isolating specific failure modes. To address this, we introduce OpenExempt, a framework and benchmark for diagnostic evaluation of legal reasoning. The OpenExempt Framework uses expert-crafted symbolic representations of U.S. Bankruptcy Code statutes to dynamically generate a large space of natural language reasoning tasks and their machine-computable solutions on demand. This gives users fine-grained control over task complexity and scope, allowing individual reasoning skills to be probed in isolation. Using this system, we construct the OpenExempt Benchmark, a diagnostic benchmark for legal reasoning with 9,765 samples across nine evaluation suites designed to carefully probe model capabilities. Experiments on 13 diverse language models reveal sharp performance cliffs that emerge only under longer reasoning paths and in the presence of obfuscating statements. We release the framework and benchmark publicly to support research aimed at understanding and improving the next generation of reasoning systems.

翻译：推理基准在语言模型的进步中发挥了关键作用。然而，严格的评估仍然是一个重大挑战，因为静态的问答对仅能提供性能的快照，将复杂行为压缩为单一的准确率指标。这一局限性在复杂且受规则约束的领域（如法律）中尤为突出，现有基准构建成本高昂，且不适用于隔离特定的故障模式。为解决此问题，我们提出了OpenExempt，一个用于法律推理诊断评估的框架与基准。OpenExempt框架利用专家构建的美国破产法典法规符号化表示，动态生成大规模的自然语言推理任务及其机器可计算的解决方案，实现按需生成。这使用户能够精细控制任务的复杂性和范围，从而能够单独探究特定的推理技能。基于此系统，我们构建了OpenExempt基准，这是一个包含9,765个样本、涵盖九个评估套件的法律推理诊断基准，旨在细致探究模型能力。对13个不同语言模型的实验揭示了仅在较长推理路径和存在混淆性陈述时才会出现的显著性能陡降现象。我们公开发布该框架与基准，以支持旨在理解和改进下一代推理系统的研究。