Generative AI holds immense promise in addressing global healthcare access challenges, with numerous innovative applications now ready for use across various healthcare domains. However, a significant barrier to the widespread adoption of these domain-specific AI solutions is the lack of robust safety mechanisms to effectively manage issues such as hallucination, misinformation, and ensuring truthfulness. Left unchecked, these risks can compromise patient safety and erode trust in healthcare AI systems. While general-purpose frameworks like Llama Guard are useful for filtering toxicity and harmful content, they do not fully address the stringent requirements for truthfulness and safety in healthcare contexts. This paper examines the unique safety and security challenges inherent to healthcare AI, particularly the risk of hallucinations, the spread of misinformation, and the need for factual accuracy in clinical settings. I propose enhancements to existing guardrails frameworks, such as Nvidia NeMo Guardrails, to better suit healthcare-specific needs. By strengthening these safeguards, I aim to ensure the secure, reliable, and accurate use of AI in healthcare, mitigating misinformation risks and improving patient safety.

翻译：生成式人工智能在应对全球医疗可及性挑战方面展现出巨大潜力，目前已有众多创新应用准备就绪，可部署于各类医疗领域。然而，这些领域专用AI解决方案广泛采用的一个主要障碍，在于缺乏健全的安全机制来有效管理幻觉、错误信息等问题，并确保信息真实性。若不加以控制，这些风险可能危及患者安全，并削弱对医疗AI系统的信任。尽管通用框架（如Llama Guard）在过滤毒性及有害内容方面具有实用性，但它们未能完全满足医疗场景中对真实性与安全性的严格要求。本文探讨了医疗AI固有的独特安全挑战，特别是幻觉风险、错误信息传播以及临床环境中对事实准确性的需求。我提出对现有护栏框架（如Nvidia NeMo Guardrails）的增强方案，以更好地适应医疗领域的特定需求。通过强化这些安全措施，旨在确保AI在医疗领域的安全、可靠与精准应用，从而降低错误信息风险并提升患者安全。