The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance, enhancing patient safety by identifying potential risks associated with medications, facilitating early detection of adverse events, and guiding regulatory decision-making. Traditional ADE detection methods are reliable but slow, not easily adaptable to large-scale operations, and offer limited information. With the exponential increase in data sources like social media content, biomedical literature, and Electronic Medical Records (EMR), extracting relevant ADE-related information from these unstructured texts is imperative. Previous ADE mining studies have focused on text-based methodologies, overlooking visual cues, limiting contextual comprehension, and hindering accurate interpretation. To address this gap, we present a MultiModal Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual information with visual aids. Additionally, we introduce a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events. Using our MMADE dataset, we showcase the significance of integrating visual cues from images to enhance overall performance. This approach holds promise for patient safety, ADE awareness, and healthcare accessibility, paving the way for further exploration in personalized healthcare.

翻译：药物不良事件（ADE）挖掘在药物警戒中至关重要，它通过识别与药物相关的潜在风险、促进不良事件的早期检测以及指导监管决策，从而提升患者安全。传统的ADE检测方法虽然可靠，但速度较慢，不易适应大规模操作，且提供的信息有限。随着社交媒体内容、生物医学文献和电子病历（EMR）等数据源的指数级增长，从这些非结构化文本中提取相关的ADE信息变得至关重要。以往的ADE挖掘研究主要集中于基于文本的方法，忽略了视觉线索，限制了上下文理解，并阻碍了准确解读。为弥补这一不足，我们提出了一个多模态药物不良事件（MMADE）检测数据集，将ADE相关的文本信息与视觉辅助材料相结合。此外，我们引入了一个框架，该框架利用LLM和VLM的能力进行ADE检测，通过生成描述ADE的医学图像的详细说明，帮助医疗专业人员从视觉上识别不良事件。使用我们的MMADE数据集，我们展示了整合图像中的视觉线索以提升整体性能的重要性。这一方法对患者安全、ADE认知和医疗可及性具有积极意义，为个性化医疗的进一步探索铺平了道路。