As LLMs gain adoption in high-stakes domains like mental health, domain experts are increasingly consulted to provide input into policies governing their behavior. From an observation of 19 policymaking workshops with 9 experts over 15 weeks, we identified opportunities to better support rapid experimentation, feedback, and iteration for collaborative policy design processes. We present PolicyPad, an interactive system that facilitates the emerging practice of LLM policy prototyping by drawing from established UX prototyping practices, including heuristic evaluation and storyboarding. Using PolicyPad, policy designers can collaborate on drafting a policy in real time while independently testing policy-informed model behavior with usage scenarios. We evaluate PolicyPad through workshops with 8 groups of 22 domain experts in mental health and law, finding that PolicyPad enhanced collaborative dynamics during policy design, enabled tight feedback loops, and led to novel policy contributions. Overall, our work paves expert-informed paths for advancing AI alignment and safety.

翻译：随着大型语言模型（LLM）在心理健康等高风险领域得到应用，领域专家越来越多地被邀请参与制定其行为管理策略。通过对9位专家在15周内进行的19次策略制定研讨会的观察，我们发现了在协作式策略设计过程中更好地支持快速实验、反馈与迭代的机遇。本文提出PolicyPad——一个借鉴启发式评估与故事板等成熟用户体验原型设计方法的交互式系统，旨在促进新兴的LLM策略原型设计实践。通过PolicyPad，策略设计者可以实时协作起草策略，同时利用使用场景独立测试基于策略的模型行为。我们通过与心理健康和法律领域的22位专家组成的8个小组进行研讨会评估PolicyPad，发现该系统增强了策略设计过程中的协作动力，实现了紧密的反馈循环，并催生了新颖的策略方案。总体而言，本研究为推进人工智能对齐与安全开辟了专家参与的新路径。