Monocular depth estimation is an important task with rapid progress, but how to evaluate it remains an open question, as evidenced by a lack of standardization in existing literature and a large selection of evaluation metrics whose trade-offs and behaviors are not well understood. This paper contributes a novel, quantitative analysis of existing metrics in terms of their sensitivity to various types of perturbations of ground truth, emphasizing comparison to human judgment. Our analysis reveals that existing metrics are severely under-sensitive to curvature perturbation such as making flat surfaces wavy. To remedy this, we introduce a new metric based on relative surface normals, along with new depth visualization tools and a principled method to create composite metrics with better human alignment. Code and data are available at: https://github.com/princeton-vl/evalmde.

翻译：单目深度估计是一项进展迅速的重要任务，但如何评估它仍然是一个悬而未决的问题，这体现在现有文献缺乏标准化，以及存在大量评估指标可供选择，而这些指标的权衡与行为尚未得到充分理解。本文对现有指标进行了新颖的定量分析，重点考察其对地面真值各类扰动的敏感性，并强调与人类判断进行比较。我们的分析表明，现有指标对曲率扰动（例如使平坦表面产生波浪形变）严重不敏感。为弥补这一缺陷，我们引入了一种基于相对表面法向的新指标，同时提供了新的深度可视化工具以及一种创建与人类判断更一致的综合指标的规范方法。代码与数据可在以下网址获取：https://github.com/princeton-vl/evalmde。