As Large Language Models and Natural Language Processing (NLP) technology rapidly develops and spreads into daily life, it becomes crucial to anticipate how its use could harm people. One problem that has received a lot of attention in recent years is that this technology has displayed harmful biases in its behavior. Although a lot of effort has been invested in assessing and mitigating these biases, our methods of measuring the biases of NLP models have serious problems (e.g., it is often unclear what they actually measure). In this paper, we provide an interdisciplinary approach to discussing the issue of NLP model bias by adopting the lens of psychometrics -- a field specialized in the measurement of concepts like bias that are not directly observable. In particular, we will explore two central notions from psychometrics, the construct validity and the reliability of measurement tools, and discuss how they can be applied in the context of measuring model bias. Our goal is to provide NLP practitioners with methodological tools for designing better bias measures, and to inspire them more generally to explore tools from psychometrics when working on bias measurement tools.

翻译：随着大语言模型与自然语言处理技术的快速发展并渗透日常生活，预见其使用可能对人类造成的伤害变得至关重要。近年来备受关注的问题之一是，该技术在其行为中表现出有害偏见。尽管已有大量研究致力于评估和缓解这些偏见，但当前测量自然语言处理模型偏见的方法存在严重问题（例如，这些方法实际测量的内容往往不明确）。本文采用心理测量学——一个专门研究不可直接观测概念（如偏见）测量的领域——的视角，以跨学科方式探讨自然语言处理模型偏见问题。具体而言，我们将探究心理测量学中的两个核心概念，即测量工具的构念效度与信度，并讨论如何在模型偏见测量情境中应用这些概念。我们的目标是为自然语言处理从业者提供设计更优偏见测量方法的方法论工具，并启发他们在开发偏见测量工具时更广泛地探索心理测量学的工具。