The integration of Large Language Models (LLMs) in information retrieval has raised a critical reevaluation of fairness in the text-ranking models. LLMs, such as GPT models and Llama2, have shown effectiveness in natural language understanding tasks, and prior works (e.g., RankGPT) have also demonstrated that the LLMs exhibit better performance than the traditional ranking models in the ranking task. However, their fairness remains largely unexplored. This paper presents an empirical study evaluating these LLMs using the TREC Fair Ranking dataset, focusing on the representation of binary protected attributes such as gender and geographic location, which are historically underrepresented in search outcomes. Our analysis delves into how these LLMs handle queries and documents related to these attributes, aiming to uncover biases in their ranking algorithms. We assess fairness from both user and content perspectives, contributing an empirical benchmark for evaluating LLMs as the fair ranker.

翻译：大型语言模型（LLMs）在信息检索中的整合引发了对文本排序模型公平性的重要重新评估。诸如GPT系列模型和Llama2等LLMs已在自然语言理解任务中展现出有效性，先前研究（例如RankGPT）也表明LLMs在排序任务中比传统排序模型具有更优性能。然而，其公平性在很大程度上仍未得到充分探索。本文通过TREC公平排序数据集对这些LLMs进行实证评估，重点关注历史上在搜索结果中代表性不足的二元受保护属性（如性别和地理位置）的呈现问题。我们的分析深入探究了这些LLMs如何处理与这些属性相关的查询和文档，旨在揭示其排序算法中存在的偏见。我们从用户和内容双重视角评估公平性，为评估LLMs作为公平排序器提供了实证基准。