Recent research increasingly brings to question the appropriateness of using predictive tools in complex, real-world tasks. While a growing body of work has explored ways to improve value alignment in these tools, comparatively less work has centered concerns around the fundamental justifiability of using these tools. This work seeks to center validity considerations in deliberations around whether and how to build data-driven algorithms in high-stakes domains. Toward this end, we translate key concepts from validity theory to predictive algorithms. We apply the lens of validity to re-examine common challenges in problem formulation and data issues that jeopardize the justifiability of using predictive algorithms and connect these challenges to the social science discourse around validity. Our interdisciplinary exposition clarifies how these concepts apply to algorithmic decision making contexts. We demonstrate how these validity considerations could distill into a series of high-level questions intended to promote and document reflections on the legitimacy of the predictive task and the suitability of the data.

翻译：近期研究越来越多地质疑在复杂现实任务中使用预测工具的适当性。尽管已有大量工作探索提升此类工具价值对齐的方法，但围绕使用这些工具的基本可证明性问题的关注相对较少。本文旨在将有效性考量置于高风险领域是否及如何构建数据驱动算法的核心讨论中。为此，我们将有效性理论中的关键概念迁移至预测算法领域，运用有效性视角重新审视威胁预测算法可证明性的常见问题（如问题构建与数据偏差），并将这些问题与社会心理学中关于有效性的学术讨论相关联。我们的跨学科阐述阐明了这些概念如何适用于算法决策情境，并展示了如何将这些有效性考量提炼为一系列高层次问题，旨在促进并记录对预测任务合法性及数据适用性的反思。