The emergence of ChatGPT and other large language models (LLMs) has greatly increased interest in utilizing LLMs as therapists to support individuals struggling with mental health challenges. However, due to the lack of systematic studies, our understanding of how LLM therapists behave, i.e., ways in which they respond to clients, is significantly limited. Understanding their behavior across a wide range of clients and situations is crucial to accurately assess their capabilities and limitations in the high-risk setting of mental health, where undesirable behaviors can lead to severe consequences. In this paper, we propose BOLT, a novel computational framework to study the conversational behavior of LLMs when employed as therapists. We develop an in-context learning method to quantitatively measure the behavior of LLMs based on 13 different psychotherapy techniques including reflections, questions, solutions, normalizing, and psychoeducation. Subsequently, we compare the behavior of LLM therapists against that of high- and low-quality human therapy, and study how their behavior can be modulated to better reflect behaviors observed in high-quality therapy. Our analysis of GPT and Llama-variants reveals that these LLMs often resemble behaviors more commonly exhibited in low-quality therapy rather than high-quality therapy, such as offering a higher degree of problem-solving advice when clients share emotions, which is against typical recommendations. At the same time, unlike low-quality therapy, LLMs reflect significantly more upon clients' needs and strengths. Our analysis framework suggests that despite the ability of LLMs to generate anecdotal examples that appear similar to human therapists, LLM therapists are currently not fully consistent with high-quality care, and thus require additional research to ensure quality care.

翻译：ChatGPT及其他大型语言模型（LLMs）的兴起，极大激发了人们利用LLMs作为治疗师来帮助心理健康挑战者的兴趣。然而，由于缺乏系统性研究，我们对LLM治疗师的行为模式——即其回应客户的方式——理解十分有限。在心理健康这一高风险场景中，不良行为可能导致严重后果，因此深入理解LLM治疗师在各类客户与情境中的行为，对于准确评估其能力与局限性至关重要。本文提出BOLT，一种创新的计算框架，用于研究LLM作为治疗师时的对话行为。我们开发了一种上下文学习方法，基于13种不同心理治疗技术（包括反思、提问、解决方案、正常化及心理教育）定量测量LLM的行为。进而，我们将LLM治疗师的行为与高质量及低质量的人类治疗行为进行对比，并研究如何调控其行为以更贴近高质量治疗中观察到的行为模式。对GPT及Llama系列模型的分析表明，这些LLM的行为往往更接近低质量治疗而非高质量治疗，例如，当客户表达情绪时，它们会提供更多问题解决建议，这与典型建议相悖。同时，与低质量治疗不同，LLM对客户的需求与优势进行了显著更多的反思。我们的分析框架表明，尽管LLM能够生成看似与人类治疗师相似的案例示例，但当前LLM治疗师尚未完全与高质量护理保持一致，因此需要进一步研究以确保优质护理。