Recent works leverage LLMs to roleplay realistic social scenarios, aiding novices in practicing their social skills. However, simulating sensitive interactions, such as in mental health, is challenging. Privacy concerns restrict data access, and collecting expert feedback, although vital, is laborious. To address this, we develop Roleplay-doh, a novel human-LLM collaboration pipeline that elicits qualitative feedback from a domain-expert, which is transformed into a set of principles, or natural language rules, that govern an LLM-prompted roleplay. We apply this pipeline to enable senior mental health supporters to create customized AI patients for simulated practice partners for novice counselors. After uncovering issues in GPT-4 simulations not adhering to expert-defined principles, we also introduce a novel principle-adherence prompting pipeline which shows 30\% improvements in response quality and principle following for the downstream task. Via a user study with 25 counseling experts, we demonstrate that the pipeline makes it easy and effective to create AI patients that more faithfully resemble real patients, as judged by creators and third-party counselors.

翻译：近期研究利用大语言模型（LLMs）模拟真实社交场景，帮助新手练习社交技能。然而，模拟敏感交互（如心理健康领域）具有挑战性。隐私问题限制了数据访问，而收集专家反馈虽至关重要却十分费力。为此，我们开发了Roleplay-doh——一种新颖的人机协作流程，该流程从领域专家处获取定性反馈，并将其转化为一系列原则（即自然语言规则），用以指导基于LLM提示的角色扮演。我们将此流程应用于支持资深心理健康工作者为新手咨询师创建定制的AI患者，作为模拟练习伙伴。在发现GPT-4模拟未遵循专家定义原则的问题后，我们还引入了一种新颖的原则遵循提示流程，该流程在下游任务中使响应质量与原则遵循度提升了30%。通过对25位咨询专家开展用户研究，我们证明该流程能帮助创建者高效便捷地构建更真实模拟真实患者的AI患者，这一结论得到了创建者与第三方咨询师的一致认可。