Stance detection (SD) identifies the text position towards a target, typically labeled as favor, against, or none. We introduce Open-Target Stance Detection (OTSD), the most realistic task where targets are neither seen during training nor provided as input. We evaluate Large Language Models (LLMs) from GPT, Gemini, Llama, and Mistral families, comparing their performance to the only existing work, Target-Stance Extraction (TSE), which benefits from predefined targets. Unlike TSE, OTSD removes the dependency of a predefined list, making target generation and evaluation more challenging. We also provide a metric for evaluating target quality that correlates well with human judgment. Our experiments reveal that LLMs outperform TSE in target generation, both when the real target is explicitly and not explicitly mentioned in the text. Similarly, LLMs overall surpass TSE in stance detection for both explicit and non-explicit cases. However, LLMs struggle in both target generation and stance detection when the target is not explicit.
翻译:立场检测旨在识别文本对特定目标的立场,通常标记为支持、反对或中立。本文提出开放目标立场检测这一最贴近现实的任务场景,其中目标既未在训练中出现,也未作为输入提供。我们评估了GPT、Gemini、Llama和Mistral系列的大型语言模型,将其性能与目前唯一相关研究——基于预定义目标的靶向立场提取方法进行对比。与TSE不同,OTSD消除了对预定义目标列表的依赖,使得目标生成与评估更具挑战性。我们同时提出了一种与人类判断高度相关的目标质量评估指标。实验表明:在文本中明确提及真实目标时,LLMs在目标生成方面优于TSE;在未明确提及目标时同样如此。类似地,无论是明确还是非明确提及目标的情况,LLMs在立场检测任务上的整体表现均超越TSE。然而,当目标未明确表述时,LLMs在目标生成和立场检测两方面均存在明显不足。