Prediction-based decision-making systems are becoming increasingly prevalent in various domains. Previous studies have demonstrated that such systems are vulnerable to runaway feedback loops, e.g., when police are repeatedly sent back to the same neighborhoods regardless of the actual rate of criminal activity, which exacerbate existing biases. In practice, the automated decisions have dynamic feedback effects on the system itself that can perpetuate over time, making it difficult for short-sighted design choices to control the system's evolution. While researchers started proposing longer-term solutions to prevent adverse outcomes (such as bias towards certain groups), these interventions largely depend on ad hoc modeling assumptions and a rigorous theoretical understanding of the feedback dynamics in ML-based decision-making systems is currently missing. In this paper, we use the language of dynamical systems theory, a branch of applied mathematics that deals with the analysis of the interconnection of systems with dynamic behaviors, to rigorously classify the different types of feedback loops in the ML-based decision-making pipeline. By reviewing existing scholarly work, we show that this classification covers many examples discussed in the algorithmic fairness community, thereby providing a unifying and principled framework to study feedback loops. By qualitative analysis, and through a simulation example of recommender systems, we show which specific types of ML biases are affected by each type of feedback loop. We find that the existence of feedback loops in the ML-based decision-making pipeline can perpetuate, reinforce, or even reduce ML biases.

翻译：基于预测的决策系统在各个领域中日益普遍。先前研究表明，此类系统容易受到失控反馈循环的影响，例如，无论实际犯罪率如何，警察被反复派往同一社区，这加剧了现有偏差。在实践中，自动化决策对系统本身产生动态反馈效应，这种效应会随时间持续，使得短视的设计选择难以控制系统的发展。尽管研究人员开始提出长期解决方案以防止不良结果（如对特定群体的偏见），但这些干预措施大多依赖于特定建模假设，而目前缺乏对基于机器学习的决策系统中反馈动态的严格理论理解。在本文中，我们运用动力系统理论的语言——应用数学的一个分支，专注于处理具有动态行为的系统间互联分析——对基于机器学习的决策管道中的不同类型反馈循环进行严谨分类。通过回顾现有学术工作，我们表明这一分类涵盖了算法公平性社区中讨论的许多实例，从而为研究反馈循环提供了一个统一且原则性的框架。通过定性分析，并借助推荐系统的仿真示例，我们展示了每种类型反馈循环如何影响特定类型的机器学习偏差。我们发现，基于机器学习的决策管道中反馈循环的存在可能持续、强化甚至减少机器学习偏差。