In the past decade, Artificial Intelligence (AI) algorithms have made promising impacts to transform healthcare in all aspects. One application is to triage patients' radiological medical images based on the algorithm's binary outputs. Such AI-based prioritization software is known as computer-aided triage and notification (CADt). Their main benefit is to speed up radiological review of images with time-sensitive findings. However, as CADt devices become more common in clinical workflows, there is still a lack of quantitative methods to evaluate a device's effectiveness in saving patients' waiting times. In this paper, we present a mathematical framework based on queueing theory to calculate the average waiting time per patient image before and after a CADt device is used. We study four workflow models with multiple radiologists (servers) and priority classes for a range of AI diagnostic performance, radiologist's reading rates, and patient image (customer) arrival rates. Due to model complexity, an approximation method known as the Recursive Dimensionality Reduction technique is applied. We define a performance metric to measure the device's time-saving effectiveness. A software tool is developed to simulate clinical workflow of image review/interpretation, to verify theoretical results, and to provide confidence intervals of the performance metric we defined. It is shown quantitatively that a triage device is more effective in a busy, short-staffed setting, which is consistent with our clinical intuition and simulation results. Although this work is motivated by the need for evaluating CADt devices, the framework we present in this paper can be applied to any algorithm that prioritizes customers based on its binary outputs.

翻译：过去十年中，人工智能（AI）算法在全方位改善医疗保健方面展现出变革性影响。其中一个应用是基于算法的二分类输出对患者放射医学影像进行分诊。此类基于AI的优先级排序软件被称为计算机辅助分诊与通知（CADt）系统，其主要优势在于加速对存在时间敏感性发现的影像的放射学审核。然而，随着CADt设备在临床工作流程中日益普及，目前仍缺乏定量评估此类设备在节省患者等待时间方面有效性的方法。本文提出一个基于排队论的数学框架，用于计算使用CADt设备前后每张患者影像的平均等待时间。我们研究了包含多位放射科医生（服务台）及优先级的四种工作流模型，覆盖不同AI诊断性能、放射科医生阅读速率及患者影像（顾客）到达速率。针对模型复杂性，采用递归降维技术这一近似方法。我们定义了一个性能指标以衡量设备的节省时间有效性，并开发了软件工具模拟影像审核与解读的临床工作流，验证理论结果，同时提供所定义性能指标的置信区间。定量结果表明，分诊设备在繁忙且人手不足的环境下更为有效，这与临床直觉及模拟结果一致。尽管本研究源于CADt设备的评估需求，但本文提出的框架可应用于任何基于二分类输出对顾客进行优先级排序的算法。