Large language models (LLMs) often reflect real-world biases, leading to efforts to mitigate these effects and make the models unbiased. Achieving this goal requires defining clear criteria for an unbiased state, with any deviation from these criteria considered biased. Some studies define an unbiased state as equal treatment across diverse demographic groups, aiming for balanced outputs from LLMs. However, differing perspectives on equality and the importance of pluralism make it challenging to establish a universal standard. Alternatively, other approaches propose using fact-based criteria for more consistent and objective evaluations, though these methods have not yet been fully applied to LLM bias assessments. Thus, there is a need for a metric with objective criteria that offers a distinct perspective from equality-based approaches. Motivated by this need, we introduce a novel metric to assess bias using fact-based criteria and real-world statistics. In this paper, we conducted a human survey demonstrating that humans tend to perceive LLM outputs more positively when they align closely with real-world demographic distributions. Evaluating various LLMs with our proposed metric reveals that model bias varies depending on the criteria used, highlighting the need for multi-perspective assessment.

翻译：大型语言模型（LLMs）常反映现实世界中的偏见，这促使人们努力减轻这些影响并使模型变得无偏见。实现这一目标需要为无偏见状态定义明确标准，任何偏离这些标准的情况均被视为存在偏见。部分研究将无偏见状态定义为对不同人口群体给予平等对待，旨在使LLMs的输出达到平衡。然而，由于对平等的不同理解以及多元化的重要性，建立普适标准面临挑战。另一些方法则提出使用基于事实的标准进行更一致和客观的评估，尽管这些方法尚未完全应用于LLM的偏见评估。因此，需要一种具备客观标准的度量方法，以提供不同于平等主义视角的独特观点。基于此需求，我们引入了一种利用基于事实的标准和现实世界统计数据评估偏见的新度量方法。本文通过人工调查表明，当LLM的输出与现实世界人口分布高度吻合时，人类倾向于对其给予更积极的评价。使用我们提出的度量方法评估多种LLMs发现，模型偏见会因采用的标准不同而变化，这凸显了多视角评估的必要性。