The aggregation of multiple opinions plays a crucial role in decision-making, such as in hiring and loan review, and in labeling data for supervised learning. Although majority voting and existing opinion aggregation models are effective for simple tasks, they are inappropriate for tasks without objectively true labels in which disagreements may occur. In particular, when voter attributes such as gender or race introduce bias into opinions, the aggregation results may vary depending on the composition of voter attributes. A balanced group of voters is desirable for fair aggregation results but may be difficult to prepare. In this study, we consider methods to achieve fair opinion aggregation based on voter attributes and evaluate the fairness of the aggregated results. To this end, we consider an approach that combines opinion aggregation models such as majority voting and the Dawid and Skene model (D&S model) with fairness options such as sample weighting. To evaluate the fairness of opinion aggregation, probabilistic soft labels are preferred over discrete class labels. First, we address the problem of soft label estimation without considering voter attributes and identify some issues with the D&S model. To address these limitations, we propose a new Soft D&S model with improved accuracy in estimating soft labels. Moreover, we evaluated the fairness of an opinion aggregation model, including Soft D&S, in combination with different fairness options using synthetic and semi-synthetic data. The experimental results suggest that the combination of Soft D&S and data splitting as a fairness option is effective for dense data, whereas weighted majority voting is effective for sparse data. These findings should prove particularly valuable in supporting decision-making by human and machine-learning models with balanced opinion aggregation.

翻译：多意见聚合在决策过程中扮演着关键角色，例如在招聘和贷款审查中，以及为监督学习标注数据时。尽管多数投票和现有的意见聚合模型对于简单任务有效，但它们不适用于没有客观真实标签且可能出现分歧的任务。特别是，当性别或种族等选民属性引入意见偏差时，聚合结果可能因选民属性的构成而不同。为实现公平的聚合结果，理想情况下需要平衡的选民群体，但这一条件可能难以满足。在本研究中，我们探讨了基于选民属性实现公平意见聚合的方法，并评估了聚合结果的公平性。为此，我们考虑了一种结合意见聚合模型（如多数投票和Dawid-Skene模型（D&S模型））与公平选项（如样本加权）的方法。为评估意见聚合的公平性，概率软标签比离散类别标签更受青睐。首先，我们研究了不考虑选民属性时软标签估计的问题，并指出了D&S模型的一些不足。为克服这些局限，我们提出了一个新的Soft D&S模型，提升了软标签估计的准确性。此外，我们结合不同的公平选项，使用合成数据和半合成数据评估了包括Soft D&S在内的意见聚合模型的公平性。实验结果表明，对于密集数据，Soft D&S与数据拆分（作为公平选项）的组合效果显著；而对于稀疏数据，加权多数投票则更为有效。这些发现对于通过平衡意见聚合来辅助人类和机器学习模型的决策具有重要价值。