AI compliance is becoming increasingly critical as AI systems grow more powerful and pervasive. Yet the rapid expansion of AI policies creates substantial burdens for resource-constrained practitioners lacking policy expertise. Existing approaches typically address one policy at a time, making multi-policy compliance costly. We present PASTA, a scalable compliance tool integrating four innovations: (1) a comprehensive model-card format supporting descriptive inputs across development stages; (2) a policy normalization scheme; (3) an efficient LLM-powered pairwise evaluation engine with cost-saving strategies; and (4) an interface delivering interpretable evaluations via compliance heatmaps and actionable recommendations. Expert evaluation shows PASTA's judgments closely align with human experts ($ρ\geq .626$). The system evaluates five major policies in under two minutes at approximately \$3. A user study (N = 12) confirms practitioners found outputs easy-to-understand and actionable, introducing a novel framework for scalable automated AI governance.
翻译:AI合规正变得日益关键,随着AI系统变得更加强大和普及。然而,政策的快速扩张给缺乏政策专业知识、资源受限的从业者带来了巨大负担。现有方法通常一次仅处理一项政策,使得多政策合规成本高昂。我们提出了PASTA,一个整合了四项创新的可扩展合规工具:(1)一种全面的模型卡片格式,支持开发各阶段的描述性输入;(2)一套政策规范化方案;(3)一个高效的基于大语言模型的成对评估引擎,并包含成本节约策略;(4)一个通过合规热图与可操作建议提供可解释评估的界面。专家评估表明,PASTA的评判与人类专家高度一致(ρ ≥ .626)。该系统评估五项主要政策耗时不到两分钟,成本约3美元。一项用户研究(N = 12)证实,从业者认为输出结果易于理解且可操作,为可扩展的自动化AI治理引入了一种新颖框架。