Search engines decide what we see for a given search query. Since many people are exposed to information through search engines, it is fair to expect that search engines are neutral. However, search engine results do not necessarily cover all the viewpoints of a search query topic, and they can be biased towards a specific view since search engine results are returned based on relevance, which is calculated using many features and sophisticated algorithms where search neutrality is not necessarily the focal point. Therefore, it is important to evaluate the search engine results with respect to bias. In this work we propose novel web search bias evaluation measures which take into account the rank and relevance. We also propose a framework to evaluate web search bias using the proposed measures and test our framework on two popular search engines based on 57 controversial query topics such as abortion, medical marijuana, and gay marriage. We measure the stance bias (in support or against), as well as the ideological bias (conservative or liberal). We observe that the stance does not necessarily correlate with the ideological leaning, e.g. a positive stance on abortion indicates a liberal leaning but a positive stance on Cuba embargo indicates a conservative leaning. Our experiments show that neither of the search engines suffers from stance bias. However, both search engines suffer from ideological bias, both favouring one ideological leaning to the other, which is more significant from the perspective of polarisation in our society.

翻译：搜索引擎决定我们针对特定查询所看到的内容。由于许多人通过搜索引擎获取信息，因此合理期望搜索引擎保持中立。然而，搜索引擎结果未必涵盖查询主题的所有观点，且可能偏向某一特定立场，因为搜索结果基于相关性返回，而相关性通过多种特征和复杂算法计算得出，且搜索中立性并非其核心关注点。因此，评估搜索引擎结果的偏差至关重要。本文提出了新颖的网络搜索偏差评估指标，这些指标综合考量了排名与相关性。我们还提出一个框架，利用所提指标评估网络搜索偏差，并基于57个争议性查询主题（如堕胎、医用大麻和同性婚姻）对两个主流搜索引擎进行测试。我们测量了立场偏差（支持或反对）以及意识形态偏差（保守派或自由派）。研究发现，立场与意识形态倾向并不必然相关，例如，对堕胎的积极立场暗示自由派倾向，但对古巴禁运的积极立场则暗示保守派倾向。实验表明，两个搜索引擎均不存在立场偏差，但都存在意识形态偏差，即各自偏向某一意识形态倾向，这从社会两极化的视角来看更为显著。