The increasing reliance on AI-driven solutions, particularly Large Language Models (LLMs) like the GPT series, for information retrieval highlights the critical need for their factuality and fairness, especially amidst the rampant spread of misinformation and disinformation online. Our study evaluates the factual accuracy, stability, and biases in widely adopted GPT models, including GPT-3.5 and GPT-4, contributing to reliability and integrity of AI-mediated information dissemination. We introduce 'Global-Liar,' a dataset uniquely balanced in terms of geographic and temporal representation, facilitating a more nuanced evaluation of LLM biases. Our analysis reveals that newer iterations of GPT models do not always equate to improved performance. Notably, the GPT-4 version from March demonstrates higher factual accuracy than its subsequent June release. Furthermore, a concerning bias is observed, privileging statements from the Global North over the Global South, thus potentially exacerbating existing informational inequities. Regions such as Africa and the Middle East are at a disadvantage, with much lower factual accuracy. The performance fluctuations over time suggest that model updates may not consistently benefit all regions equally. Our study also offers insights into the impact of various LLM configuration settings, such as binary decision forcing, model re-runs and temperature, on model's factuality. Models constrained to binary (true/false) choices exhibit reduced factuality compared to those allowing an 'unclear' option. Single inference at a low temperature setting matches the reliability of majority voting across various configurations. The insights gained highlight the need for culturally diverse and geographically inclusive model training and evaluation. This approach is key to achieving global equity in technology, distributing AI benefits fairly worldwide.

翻译：随着对AI驱动解决方案（特别是GPT系列等大型语言模型）在信息检索中的依赖日益增加，确保其事实性和公平性变得至关重要，尤其是在虚假信息和错误信息网上泛滥的背景下。本研究评估了广泛采用的GPT模型（包括GPT-3.5和GPT-4）的事实准确性、稳定性和偏见，以促进AI中介信息传播的可靠性和完整性。我们提出了"Global-Liar"数据集，该数据集在地理和时间表征上具有独特的平衡性，便于对LLM偏见进行更细致的评估。分析显示，GPT模型的新版本并不总是等同于性能提升。值得注意的是，3月发布的GPT-4版本比其后续6月版本展现出更高的事实准确性。此外，观察到一种令人担忧的偏见，即偏向全球北方而非全球南方的陈述，这可能加剧现有的信息不平等。非洲和中东等地区处于劣势，事实准确性远低于其他地区。随时间变化的性能波动表明，模型更新可能并非始终均等地惠及所有地区。我们的研究还提供了关于各种LLM配置设置（如二进制决策强制、模型重运行和温度）对模型事实性影响的见解。与允许"不明确"选项的模型相比，被限制为二进制（真/假）选择的模型表现出较低的事实准确性。在低温度设置下进行单次推理的可靠性，可与各种配置下的多数投票相匹配。这些发现强调了需要具有文化多样性和地理包容性的模型训练与评估。这一方法实现在技术领域达成全球公平、在全球范围内公平分配AI益处的关键。