High-quality psychological counseling is crucial for mental health worldwide, and timely evaluation is vital for ensuring its effectiveness. However, obtaining professional evaluation for each counseling session is expensive and challenging. Existing methods that rely on self or third-party manual reports to assess the quality of counseling suffer from subjective biases and limitations of time-consuming. To address above challenges, this paper proposes an innovative and efficient automatic approach using large language models (LLMs) to evaluate the working alliance in counseling conversations. We collected a comprehensive counseling dataset and conducted multiple third-party evaluations based on therapeutic relationship theory. Our LLM-based evaluation, combined with our guidelines, shows high agreement with human evaluations and provides valuable insights into counseling scripts. This highlights the potential of LLMs as supervisory tools for psychotherapists. By integrating LLMs into the evaluation process, our approach offers a cost-effective and dependable means of assessing counseling quality, enhancing overall effectiveness.

翻译：高质量的心理咨询对于全球心理健康至关重要，而及时的评估是确保其有效性的关键。然而，对每次咨询进行专业评估既昂贵又困难。现有依赖自我报告或第三方人工报告评估咨询质量的方法存在主观偏差和耗时限制等问题。针对上述挑战，本文提出一种创新且高效的自动评估方法，利用大语言模型对咨询对话中的工作同盟进行评价。我们收集了全面的咨询数据集，并基于治疗关系理论开展了多项第三方评估。结合我们制定的评估准则，基于大语言模型的评估方法与人类评估结果具有高度一致性，为咨询文本提供了有价值的见解。这凸显了大语言模型作为心理治疗师监督工具的潜力。通过将大语言模型融入评估流程，我们的方法提供了一种经济可行且可靠的咨询质量评估手段，进而提升整体咨询效果。