The emergence of large language models (LLMs) like ChatGPT has increased interest in their use as therapists to address mental health challenges and the widespread lack of access to care. However, experts have emphasized the critical need for systematic evaluation of LLM-based mental health interventions to accurately assess their capabilities and limitations. Here, we propose BOLT, a proof-of-concept computational framework to systematically assess the conversational behavior of LLM therapists. We quantitatively measure LLM behavior across 13 psychotherapeutic approaches with in-context learning methods. Then, we compare the behavior of LLMs against high- and low-quality human therapy. Our analysis based on Motivational Interviewing therapy reveals that LLMs often resemble behaviors more commonly exhibited in low-quality therapy rather than high-quality therapy, such as offering a higher degree of problem-solving advice when clients share emotions. However, unlike low-quality therapy, LLMs reflect significantly more upon clients' needs and strengths. Our findings caution that LLM therapists still require further research for consistent, high-quality care.

翻译：以ChatGPT为代表的大型语言模型（LLMs）的出现，引发了人们对其作为治疗师应对心理健康挑战、缓解普遍存在的医疗服务可及性不足问题的兴趣。然而，专家们强调，亟需对基于LLM的心理健康干预措施进行系统性评估，以准确判断其能力与局限。本文提出BOLT，一个概念验证性的计算框架，用于系统性评估LLM治疗师的对话行为。我们采用上下文学习方法，在13种心理治疗取向上对LLM行为进行了定量测量。随后，我们将LLM的行为与高质量及低质量的人类治疗进行了比较。基于动机性访谈治疗的分析表明，LLM的行为往往更类似于低质量治疗中常见的模式，而非高质量治疗。例如，当来访者表达情绪时，LLM会提供更高程度的问题解决建议。然而，与低质量治疗不同的是，LLM对来访者的需求和优势表现出显著更多的反思。我们的研究结果提示，LLM治疗师要提供稳定、高质量的治疗，仍需要进一步的研究。