Stance Detection is concerned with identifying the attitudes expressed by an author towards a target of interest. This task spans a variety of domains ranging from social media opinion identification to detecting the stance for a legal claim. However, the framing of the task varies within these domains, in terms of the data collection protocol, the label dictionary and the number of available annotations. Furthermore, these stance annotations are significantly imbalanced on a per-topic and inter-topic basis. These make multi-domain stance detection a challenging task, requiring standardization and domain adaptation. To overcome this challenge, we propose $\textbf{T}$opic $\textbf{E}$fficient $\textbf{St}$anc$\textbf{E}$ $\textbf{D}$etection (TESTED), consisting of a topic-guided diversity sampling technique and a contrastive objective that is used for fine-tuning a stance classifier. We evaluate the method on an existing benchmark of $16$ datasets with in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics, experiments. The results show that our method outperforms the state-of-the-art with an average of $3.5$ F1 points increase in-domain, and is more generalizable with an averaged increase of $10.2$ F1 on out-of-domain evaluation while using $\leq10\%$ of the training data. We show that our sampling technique mitigates both inter- and per-topic class imbalances. Finally, our analysis demonstrates that the contrastive learning objective allows the model a more pronounced segmentation of samples with varying labels.
翻译:立场检测旨在识别作者针对特定目标所表达的态度。该任务涵盖从社交媒体观点识别到法律主张立场检测等多个领域。然而,这些领域内任务的框架存在差异,涉及数据收集协议、标签字典以及可用标注数量。此外,这些立场标注在主题内部及主题之间均存在显著的不平衡性。这使得多领域立场检测成为一项具有挑战性的任务,需要标准化和领域自适应。为应对这一挑战,我们提出了$\textbf{T}$opic $\textbf{E}$fficient $\textbf{St}$anc$\textbf{E}$ $\textbf{D}$etection(TESTED),该方法包括主题引导的多样性采样技术以及用于微调立场分类器的对比学习目标。我们在包含$16$个数据集的现有基准上进行了域内(即所有主题均已见过)和域外(即未见主题)实验评估。结果表明,我们的方法在域内平均提升了$3.5$个F1分数点,超越了当前最优水平;在域外评估中,平均提升了$10.2$个F1分数点,且仅使用$\leq10\%$的训练数据时,泛化能力更强。我们证明了采样技术能够缓解主题间和主题内的类别不平衡问题。最后,我们的分析表明,对比学习目标使模型能够更显著地区分不同标签的样本。