The gold standard in human-AI collaboration is complementarity -- when combined performance exceeds both the human and algorithm alone. We investigate this challenge in binary classification settings where the goal is to maximize 0-1 accuracy. Given two or more agents who can make calibrated probabilistic predictions, we show a "No Free Lunch"-style result. Any deterministic collaboration strategy (a function mapping calibrated probabilities into binary classifications) that does not essentially always defer to the same agent will sometimes perform worse than the least accurate agent. In other words, complementarity cannot be achieved "for free." The result does suggest one model of collaboration with guarantees, where one agent identifies "obvious" errors of the other agent. We also use the result to understand the necessary conditions enabling the success of other collaboration techniques, providing guidance to human-AI collaboration.
翻译:人机协作的黄金标准是互补性——即组合性能超越人类和算法单独的性能。我们在二元分类场景中研究这一挑战,其目标是最大化0-1准确率。给定两个或多个能够做出校准概率预测的智能体,我们证明了一个"无免费午餐"式的结果:任何确定性协作策略(将校准概率映射为二元分类的函数),若本质上不总是遵从同一智能体,则有时会表现不如最不准确的智能体。换言之,互补性无法"免费"实现。该结果确实提出了一种具有保证的协作模型,即一个智能体识别另一个智能体的"明显"错误。我们还利用该结果理解其他协作技术成功的必要条件,为人机协作提供指导。