The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance, enhancing patient safety by identifying potential risks associated with medications, facilitating early detection of adverse events, and guiding regulatory decision-making. Traditional ADE detection methods are reliable but slow, not easily adaptable to large-scale operations, and offer limited information. With the exponential increase in data sources like social media content, biomedical literature, and Electronic Medical Records (EMR), extracting relevant ADE-related information from these unstructured texts is imperative. Previous ADE mining studies have focused on text-based methodologies, overlooking visual cues, limiting contextual comprehension, and hindering accurate interpretation. To address this gap, we present a MultiModal Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual information with visual aids. Additionally, we introduce a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events. Using our MMADE dataset, we showcase the significance of integrating visual cues from images to enhance overall performance. This approach holds promise for patient safety, ADE awareness, and healthcare accessibility, paving the way for further exploration in personalized healthcare.

翻译：药物不良事件挖掘在药物警戒中至关重要，它通过识别药物相关潜在风险、促进不良事件早期检测以及指导监管决策，从而提升患者安全。传统的药物不良事件检测方法虽可靠但速度较慢，难以适应大规模操作，且提供的信息有限。随着社交媒体内容、生物医学文献和电子病历等数据源的指数级增长，从这些非结构化文本中提取相关药物不良事件信息变得尤为迫切。以往的药物不良事件挖掘研究主要集中于基于文本的方法，忽视了视觉线索，限制了上下文理解，并阻碍了准确解读。为填补这一空白，我们提出了一个多模态药物不良事件检测数据集，将药物不良事件相关的文本信息与视觉辅助材料相结合。此外，我们引入了一个框架，利用大语言模型和视觉语言模型的能力，通过生成描述药物不良事件医学图像的详细说明，帮助医疗专业人员直观识别不良事件。基于我们的多模态药物不良事件检测数据集，我们展示了整合图像视觉线索以提升整体性能的重要性。该方法在患者安全、药物不良事件认知和医疗可及性方面具有广阔前景，为个性化医疗的进一步探索铺平了道路。