We focus on the problem of ranking $N$ objects starting from a set of noisy pairwise comparisons provided by a crowd of unequal workers, each worker being characterized by a specific degree of reliability, which reflects her ability to rank pairs of objects. More specifically, we assume that objects are endowed with intrinsic qualities and that the probability with which an object is preferred to another depends both on the difference between the qualities of the two competitors and on the reliability of the worker. We propose QUITE, a non-adaptive ranking algorithm that jointly estimates workers' reliabilities and qualities of objects. Performance of QUITE is compared in different scenarios against previously proposed algorithms. Finally, we show how QUITE can be naturally made adaptive.

翻译：我们聚焦于从一组由不同可靠度的工作者提供的含噪两两比较中，对N个对象进行排序的问题。每个工作者具有特定的可靠度，反映其排序对象对的能力。具体而言，我们假设对象具有内在质量，且一个对象被偏好于另一个的概率，既取决于两者质量之差，也取决于工作者的可靠度。我们提出QUITE，一种非自适应排序算法，能联合估计工作者的可靠度与对象的质量。在不同场景下，将QUITE的性能与先前提出的算法进行比较。最后，我们展示如何自然地将QUITE扩展为自适应算法。