Early warning systems (EWS) are prediction algorithms that have recently taken a central role in efforts to improve graduation rates in public schools across the US. These systems assist in targeting interventions at individual students by predicting which students are at risk of dropping out. Despite significant investments and adoption, there remain significant gaps in our understanding of the efficacy of EWS. In this work, we draw on nearly a decade's worth of data from a system used throughout Wisconsin to provide the first large-scale evaluation of the long-term impact of EWS on graduation outcomes. We present evidence that risk assessments made by the prediction system are highly accurate, including for students from marginalized backgrounds. Despite the system's accuracy and widespread use, we find no evidence that it has led to improved graduation rates. We surface a robust statistical pattern that can explain why these seemingly contradictory insights hold. Namely, environmental features, measured at the level of schools, contain significant signal about dropout risk. Within each school, however, academic outcomes are essentially independent of individual student performance. This empirical observation indicates that assigning all students within the same school the same probability of graduation is a nearly optimal prediction. Our work provides an empirical backbone for the robust, qualitative understanding among education researchers and policy-makers that dropout is structurally determined. The primary barrier to improving outcomes lies not in identifying students at risk of dropping out within specific schools, but rather in overcoming structural differences across different school districts. Our findings indicate that we should carefully evaluate the decision to fund early warning systems without also devoting resources to interventions tackling structural barriers.

翻译：早期预警系统是预测算法，近期在美国公立学校提升毕业率的举措中占据核心地位。这些系统通过预测哪些学生面临辍学风险，协助针对个体学生进行干预。尽管投入了大量资源并得到广泛采用，但我们对早期预警系统有效性的理解仍存在显著空白。本研究利用威斯康星州范围内一个系统近十年的数据，首次大规模评估了早期预警系统对毕业结果的长期影响。我们提供的证据表明，该预测系统的风险评估高度准确，包括对来自边缘化背景的学生也是如此。尽管该系统准确且被广泛使用，但我们未发现其改善了毕业率的证据。我们揭示了一个稳健的统计模式，可解释为何这些看似矛盾的观点成立。即在学校层面测量的环境特征包含了关于辍学风险的显著信号。然而，在每个学校内部，学业结果与个体学生的表现基本无关。这一实证观察表明，为同一学校内所有学生分配相同的毕业概率几乎是最优预测。我们的工作为教育研究人员和政策制定者关于辍学是结构性决定的稳健定性理解提供了实证基础。改善结果的主要障碍并非在于识别特定学校内有辍学风险的学生，而在于克服不同学区间的结构性差异。我们的研究结果表明，若未同时投入资源应对结构性障碍的干预措施，应谨慎评估为早期预警系统提供资金的决策。