Being able to infer ground truth from the responses of multiple imperfect advisors is a problem of crucial importance in many decision-making applications, such as lending, trading, investment, and crowd-sourcing. In practice, however, gathering answers from a set of advisors has a cost. Therefore, finding an advisor selection strategy that retrieves a reliable answer and maximizes the overall utility is a challenging problem. To address this problem, we propose a novel strategy for optimally selecting a set of advisers in a sequential binary decision-making setting, where multiple decisions need to be made over time. Crucially, we assume no access to ground truth and no prior knowledge about the reliability of advisers. Specifically, our approach considers how to simultaneously (1) select advisors by balancing the advisors' costs and the value of making correct decisions, (2) learn the trustworthiness of advisers dynamically without prior information by asking multiple advisers, and (3) make optimal decisions without access to the ground truth, improving this over time. We evaluate our algorithm through several numerical experiments. The results show that our approach outperforms two other methods that combine state-of-the-art models.

翻译：能够从多个不完美顾问的响应中推断出真实情况，是在许多决策应用中至关重要的问题，例如贷款、交易、投资和众包。然而在实践中，从一组顾问那里收集答案是有成本的。因此，找到一种既能获得可靠答案又能最大化整体效用的顾问选择策略是一个具有挑战性的问题。为解决这个问题，我们提出了一种新策略，用于在序贯二元决策设置中优化选择一组顾问，其中需要随时间做出多个决策。关键的是，我们假设无法访问真实情况，且没有关于顾问可靠性的先验知识。具体来说，我们的方法同时考虑如何：(1) 通过平衡顾问成本与做出正确决策的价值来选择顾问，(2) 在没有先验信息的情况下通过询问多个顾问动态学习顾问的可信度，以及(3) 在无法访问真实情况下做出最优决策，并随时间推移改进这一决策。我们通过多项数值实验评估了我们的算法。结果表明，我们的方法优于结合了先进模型的其他两种方法。