Intelligent agents rely on AI/ML functionalities to predict the consequence of possible actions and optimise the policy. However, the effort of the research community in addressing prediction accuracy has been so intense (and successful) that it created the illusion that the more accurate the learner prediction (or classification) the better would have been the final decision. Now, such an assumption is valid only if the (human or artificial) decision maker has complete knowledge of the utility of the possible actions. This paper argues that AI/ML community has taken so far a too unbalanced approach by devoting excessive attention to the estimation of the state (or target) probability to the detriment of accurate and reliable estimations of the utility. In particular, few evidence exists about the impact of a wrong utility assessment on the resulting expected utility of the decision strategy. This situation is creating a substantial gap between the expectations and the effective impact of AI solutions, as witnessed by recent criticisms and emphasised by the regulatory legislative efforts. This paper aims to study this gap by quantifying the sensitivity of the expected utility to the utility uncertainty and comparing it to the one due to probability estimation. Theoretical and simulated results show that an inaccurate utility assessment may as (and sometimes) more harmful than a poor probability estimation. The final recommendation to the community is then to undertake a focus shift from a pure accuracy-driven (or obsessed) approach to a more utility-aware methodology.

翻译：智能体依赖人工智能/机器学习功能来预测可能行动的后果并优化策略。然而，研究界在解决预测准确性方面投入了如此巨大的精力（且成绩斐然），以至于产生了一种错觉：学习者的预测（或分类）越准确，最终决策就会越好。事实上，这种假设仅在（人类或人工）决策者完全了解可能行动的效用时成立。本文认为，人工智能/机器学习界迄今采取了过于失衡的方法，过度关注状态（或目标）概率的估计，而损害了对效用的准确可靠估计。特别是，关于错误效用评估对决策策略预期效用影响的证据很少。这种状况正在造成人工智能解决方案的期望与实际效果之间的巨大差距，正如近期的批评所证明的那样，并因监管立法努力而更加凸显。本文旨在通过量化预期效用对效用不确定性的敏感性，并将其与概率估计导致的敏感性进行比较来研究这一差距。理论和模拟结果表明，不准确的效用评估可能（有时甚至）比糟糕的概率估计更具危害性。因此，向研究界提出的最终建议是：将关注重点从纯粹以准确性为驱动（或痴迷）的方法转向更加注重效用的方法论。