Identifying user's opinions and stances in long conversation threads on various topics can be extremely critical for enhanced personalization, market research, political campaigns, customer service, conflict resolution, targeted advertising, and content moderation. Hence, training language models to automate this task is critical. However, to train such models, gathering manual annotations has multiple challenges: 1) It is time-consuming and costly; 2) Conversation threads could be very long, increasing chances of noisy annotations; and 3) Interpreting instances where a user changes their opinion within a conversation is difficult because often such transitions are subtle and not expressed explicitly. Inspired by the recent success of large language models (LLMs) for complex natural language processing (NLP) tasks, we leverage Mistral Large and GPT-4 to automate the human annotation process on the following two tasks while also providing reasoning: i) User Stance classification, which involves labeling a user's stance of a post in a conversation on a five-point scale; ii) User Dogmatism classification, which deals with labeling a user's overall opinion in the conversation on a four-point scale. The majority voting on zero-shot, one-shot, and few-shot annotations from these two LLMs on 764 multi-user Reddit conversations helps us curate the USDC dataset. USDC is then used to finetune and instruction-tune multiple deployable small language models for the 5-class stance and 4-class dogmatism classification tasks. We make the code and dataset publicly available [https://anonymous.4open.science/r/USDC-0F7F].

翻译：在各种主题的长对话线程中识别用户的观点和立场，对于增强个性化服务、市场研究、政治竞选、客户服务、冲突解决、定向广告和内容审核都至关重要。因此，训练语言模型以自动化此任务具有重要意义。然而，为训练此类模型，收集人工标注面临多重挑战：1) 过程耗时且成本高昂；2) 对话线程可能非常长，增加了标注噪声的可能性；3) 解释用户在对话中改变观点的实例较为困难，因为此类转变通常微妙且未明确表达。受近期大语言模型在复杂自然语言处理任务中成功的启发，我们利用 Mistral Large 和 GPT-4 自动化以下两项任务的人工标注过程，并提供推理依据：i) 用户立场分类，即按五级量表标注对话中用户帖子的立场；ii) 用户独断性分类，即按四级量表标注用户在对话中的整体观点倾向。通过对这两种大语言模型在 764 个多用户 Reddit 对话上进行零样本、单样本和少样本标注的多数投票，我们构建了 USDC 数据集。随后，USDC 被用于对多个可部署的小型语言模型进行微调和指令调优，以执行 5 类立场分类和 4 类独断性分类任务。我们已公开代码和数据集 [https://anonymous.4open.science/r/USDC-0F7F]。