Risk assessment algorithms are being adopted by public sector agencies to make high-stakes decisions about human lives. Algorithms model "risk" based on individual client characteristics to identify clients most in need. However, this understanding of risk is primarily based on easily quantifiable risk factors that present an incomplete and biased perspective of clients. We conducted a computational narrative analysis of child-welfare casenotes and draw attention to deeper systemic risk factors that are hard to quantify but directly impact families and street-level decision-making. We found that beyond individual risk factors, the system itself poses a significant amount of risk where parents are over-surveilled by caseworkers and lack agency in decision-making processes. We also problematize the notion of risk as a static construct by highlighting the temporality and mediating effects of different risk, protective, systemic, and procedural factors. Finally, we draw caution against using casenotes in NLP-based systems by unpacking their limitations and biases embedded within them.

翻译：风险评估算法正被公共部门机构用于对人类生活做出高风险的决策。算法基于个体客户特征对“风险”建模，以识别最需要帮助的客户。然而，这种风险理解主要依赖于易于量化的风险因素，这些因素呈现了不完整且有偏差的客户视角。我们对儿童福利案例记录进行了计算叙事分析，并关注那些难以量化但直接影响家庭和基层决策的更深层系统性风险因素。我们发现，除个体风险因素外，系统本身也构成显著风险，例如父母被案件工作者过度监控，且在决策过程中缺乏自主权。此外，我们通过强调不同风险、保护性、系统性和程序性因素的时间性及中介效应，对风险作为静态构念的概念提出了质疑。最后，我们通过揭示案例记录中存在的局限性及嵌入其中的偏见，警示在基于NLP的系统中使用案例记录的风险。