Medication mistaking is one of the risks that can result in unpredictable consequences for patients. To mitigate this risk, we develop an automatic system that correctly identifies pill-prescription from mobile images. Specifically, we define a so-called pill-prescription matching task, which attempts to match the images of the pills taken with the pills' names in the prescription. We then propose PIMA, a novel approach using Graph Neural Network (GNN) and contrastive learning to address the targeted problem. In particular, GNN is used to learn the spatial correlation between the text boxes in the prescription and thereby highlight the text boxes carrying the pill names. In addition, contrastive learning is employed to facilitate the modeling of cross-modal similarity between textual representations of pill names and visual representations of pill images. We conducted extensive experiments and demonstrated that PIMA outperforms baseline models on a real-world dataset of pill and prescription images that we constructed. Specifically, PIMA improves the accuracy from 19.09% to 46.95% compared to other baselines. We believe our work can open up new opportunities to build new clinical applications and improve medication safety and patient care.

翻译：为了减轻这一风险,我们开发了一个自动系统,正确识别移动图像中的避孕药处方。具体地说,我们定义了一种所谓的避孕药处方匹配任务,试图将药片的图像与药片在处方中的名称相匹配。然后我们提出PIMA,这是使用神经网络图像(GNN)和对比性学习来解决目标问题的新方法之一。特别是,GNN用来学习处方文本框之间的空间相关性,从而突出含有药片名称的文本框。此外,我们采用了对比学习,以促进药片名称的文本展示和药片图像的视觉展示之间的跨模式相似性。我们进行了广泛的实验,并展示了PIMA在我们制造的药片和处方图像真实世界数据集上优于基线模型。具体地说,PIMA与其他基线相比,提高了从19.09%到46.95%的准确性。我们相信我们的工作可以打开新的机会,以建立新的临床应用,改善药物安全和病人护理。