The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while changes to standard generation parameters have minimal effect on PCT scores, prompt phrasing and fine-tuning individually and together can significantly influence results. Interestingly, fine-tuning on politically rich vs. neutral datasets does not lead to different shifts in scores. We also generalize these findings to a similar popular test called 8 Values. Humans do not change their responses to questions when prompted differently (``answer this question'' vs ``state your opinion''), or after exposure to politically neutral text, such as mathematical formulae. But the fact that the models do so raises concerns about the validity of these tests for measuring model bias, and paves the way for deeper exploration into how political and social views are encoded in LLMs.

翻译：政治罗盘测试（PCT）及类似调查常用于评估自回归大型语言模型的政治偏见。我们严格的统计实验表明，虽然标准生成参数的调整对PCT得分影响甚微，但提示措辞与微调操作单独或共同作用均能显著改变结果。值得注意的是，在政治内容丰富与中立数据集上的微调并未导致得分偏移方向的差异。我们进一步将这些发现推广至名为“8 Values”的类似流行测试中。人类在面临不同提示（如“回答此问题”与“陈述你的观点”）或接触数学公式等政治中立文本后，不会改变对问题的回答。然而模型表现出的敏感性引发了对这些测试衡量模型偏见有效性的质疑，并为深入探究政治与社会观点在大型语言模型中的编码机制开辟了新路径。