Language Models have ushered a new age of AI gaining traction within the NLP community as well as amongst the general population. AI's ability to make predictions, generations and its applications in sensitive decision-making scenarios, makes it even more important to study these models for possible biases that may exist and that can be exaggerated. We conduct a quality comparative study and establish a framework to evaluate language models under the premise of two kinds of biases: gender and race, in a professional setting. We find out that while gender bias has reduced immensely in newer models, as compared to older ones, racial bias still exists.

翻译：语言模型开启了人工智能的新时代，不仅在自然语言处理学界备受关注，也吸引了广大公众的注意。由于AI具备预测、生成能力，并且在涉及敏感决策的场景中得到应用，研究这些模型可能存在的、甚至可能被放大的偏见就显得尤为重要。我们开展了一项质量对比研究，并构建了一个框架，用以在专业场景下评估语言模型中存在的两种偏见：性别偏见与种族偏见。研究发现，尽管新模型中的性别偏见相较于旧模型已显著减少，但种族偏见依然存在。