With calls for increasing transparency, governments are releasing greater amounts of data in multiple domains including finance, education and healthcare. The efficient exploratory analysis of healthcare data constitutes a significant challenge. Key concerns in public health include the quick identification and analysis of trends, and the detection of outliers. This allows policies to be rapidly adapted to changing circumstances. We present an efficient outlier detection technique, termed PIKS (Pruned iterative-k means searchlight), which combines an iterative k-means algorithm with a pruned searchlight based scan. We apply this technique to identify outliers in two publicly available healthcare datasets from the New York Statewide Planning and Research Cooperative System, and California's Office of Statewide Health Planning and Development. We provide a comparison of our technique with three other existing outlier detection techniques, consisting of auto-encoders, isolation forests and feature bagging. We identified outliers in conditions including suicide rates, immunity disorders, social admissions, cardiomyopathies, and pregnancy in the third trimester. We demonstrate that the PIKS technique produces results consistent with other techniques such as the auto-encoder. However, the auto-encoder needs to be trained, which requires several parameters to be tuned. In comparison, the PIKS technique has far fewer parameters to tune. This makes it advantageous for fast, "out-of-the-box" data exploration. The PIKS technique is scalable and can readily ingest new datasets. Hence, it can provide valuable, up-to-date insights to citizens, patients and policy-makers. We have made our code open source, and with the availability of open data, other researchers can easily reproduce and extend our work. This will help promote a deeper understanding of healthcare policies and public health issues.
翻译:随着对透明度提升的呼声日益高涨,各国政府在金融、教育和医疗等多个领域发布的数据量持续增加。对医疗数据进行高效探索性分析是一项重大挑战。公共卫生领域的关键问题包括快速识别和分析趋势,以及检测异常值。这有助于政策能够快速适应不断变化的环境。我们提出了一种高效的异常值检测技术,称为PIKS(剪枝迭代k均值探照灯),该技术将迭代k均值算法与基于剪枝探照灯的扫描相结合。我们将该技术应用于两组公开的医疗数据集:纽约州全州规划与研究合作系统以及加利福尼亚州全州健康规划与发展办公室的数据。我们将该技术与三种现有的异常值检测技术(自编码器、孤立森林和特征装袋)进行了比较。我们识别了多种条件下的异常值,包括自杀率、免疫系统疾病、社会性住院、心肌病以及妊娠晚期。我们证明,PIKS技术产生的结果与其他技术(如自编码器)一致。然而,自编码器需要训练,这要求调整多个参数。相比之下,PIKS技术需要调整的参数少得多。这使得它在快速“开箱即用”的数据探索中具有优势。PIKS技术具有可扩展性,并能轻松吸收新数据集。因此,它能向公民、患者和政策制定者提供有价值的、最新的洞察。我们已将代码开源,并且凭借开放数据的可用性,其他研究人员能够轻松复现和扩展我们的工作。这将有助于促进对医疗政策和公共卫生问题的更深入理解。