Large Language Models (LLMs) have demonstrated remarkable capabilities in executing tasks based on natural language queries. However, these models, trained on curated datasets, inherently embody biases ranging from racial to national and gender biases. It remains uncertain whether these biases impact the performance of LLMs for certain tasks. In this study, we investigate the political biases of LLMs within the stance classification task, specifically examining whether these models exhibit a tendency to more accurately classify politically-charged stances. Utilizing three datasets, seven LLMs, and four distinct prompting schemes, we analyze the performance of LLMs on politically oriented statements and targets. Our findings reveal a statistically significant difference in the performance of LLMs across various politically oriented stance classification tasks. Furthermore, we observe that this difference primarily manifests at the dataset level, with models and prompting schemes showing statistically similar performances across different stance classification datasets. Lastly, we observe that when there is greater ambiguity in the target the statement is directed towards, LLMs have poorer stance classification accuracy.

翻译：大语言模型（LLMs）在执行基于自然语言查询的任务方面展现出卓越的能力。然而，这些模型在精心策划的数据集上训练而成，其本身内嵌了从种族、国家到性别等多种偏见。目前尚不确定这些偏见是否会影响LLMs在某些任务上的表现。在本研究中，我们探究了LLMs在立场分类任务中的政治偏见，特别检验了这些模型是否表现出更准确分类政治立场陈述的倾向。通过使用三个数据集、七个LLM以及四种不同的提示方案，我们分析了LLMs在政治导向的陈述与目标上的表现。我们的研究结果表明，LLMs在不同政治导向的立场分类任务中的表现存在统计学上的显著差异。此外，我们观察到这种差异主要体现在数据集层面，而模型和提示方案在不同立场分类数据集上表现出统计上相似的性能。最后，我们发现当陈述所指向的目标存在较大模糊性时，LLMs的立场分类准确性会降低。