Large language models (LLMs) have shown promising capabilities in healthcare analysis but face several challenges like hallucinations, parroting, and bias manifestation. These challenges are exacerbated in complex, sensitive, and low-resource domains. Therefore, in this work we introduce IC-AnnoMI, an expert-annotated motivational interviewing (MI) dataset built upon AnnoMI by generating in-context conversational dialogues leveraging LLMs, particularly ChatGPT. IC-AnnoMI employs targeted prompts accurately engineered through cues and tailored information, taking into account therapy style (empathy, reflection), contextual relevance, and false semantic change. Subsequently, the dialogues are annotated by experts, strictly adhering to the Motivational Interviewing Skills Code (MISC), focusing on both the psychological and linguistic dimensions of MI dialogues. We comprehensively evaluate the IC-AnnoMI dataset and ChatGPT's emotional reasoning ability and understanding of domain intricacies by modeling novel classification tasks employing several classical machine learning and current state-of-the-art transformer approaches. Finally, we discuss the effects of progressive prompting strategies and the impact of augmented data in mitigating the biases manifested in IC-AnnoM. Our contributions provide the MI community with not only a comprehensive dataset but also valuable insights for using LLMs in empathetic text generation for conversational therapy in supervised settings.

翻译：大语言模型（LLMs）在医疗健康分析中展现出潜力，但仍面临幻觉、鹦鹉学舌及偏见显现等挑战。这些挑战在复杂、敏感且资源匮乏的领域中尤为突出。为此，本研究引入IC-AnnoMI——一个基于AnnoMI构建、由专家标注的动机性访谈（MI）数据集，其通过利用大语言模型（特别是ChatGPT）生成上下文对话构建而成。IC-AnnoMI采用基于线索和定制信息精准设计的定向提示，综合考虑了治疗风格（共情、反思）、上下文相关性及虚假语义变化。随后，由专家严格遵循动机性访谈技能编码（MISC）标准对对话进行标注，重点关注MI对话的心理与语言维度。我们通过构建新颖的分类任务，采用多种经典机器学习方法及当前最先进的Transformer模型，对IC-AnnoMI数据集及ChatGPT的情感推理能力与领域复杂性理解进行了全面评估。最后，我们探讨了渐进式提示策略的效果以及增强数据在缓解IC-AnnoMI中显现的偏见方面的作用。本研究的贡献不仅为MI领域提供了一个综合性数据集，也为在监督环境下利用LLMs进行会话治疗中的共情文本生成提供了宝贵见解。