The expanding application of Artificial Intelligence (AI) in scientific fields presents unprecedented opportunities for discovery and innovation. However, this growth is not without risks. AI models in science, if misused, can amplify risks like creation of harmful substances, or circumvention of established regulations. In this study, we aim to raise awareness of the dangers of AI misuse in science, and call for responsible AI development and use in this domain. We first itemize the risks posed by AI in scientific contexts, then demonstrate the risks by highlighting real-world examples of misuse in chemical science. These instances underscore the need for effective risk management strategies. In response, we propose a system called SciGuard to control misuse risks for AI models in science. We also propose a red-teaming benchmark SciMT-Safety to assess the safety of different systems. Our proposed SciGuard shows the least harmful impact in the assessment without compromising performance in benign tests. Finally, we highlight the need for a multidisciplinary and collaborative effort to ensure the safe and ethical use of AI models in science. We hope that our study can spark productive discussions on using AI ethically in science among researchers, practitioners, policymakers, and the public, to maximize benefits and minimize the risks of misuse.

翻译：人工智能在科学领域的广泛应用为发现与创新带来了前所未有的机遇。然而，这种增长并非没有风险。科学领域中的AI模型若被滥用，可能加剧产生有害物质或规避现有法规等风险。本研究旨在提高人们对科学中AI滥用危险的认知，并呼吁在该领域负责任地开发和使用AI。我们首先列举了AI在科学背景下带来的风险，随后通过化学科学中真实发生的滥用案例来展示这些风险。这些实例凸显了制定有效风险管理策略的必要性。为此，我们提出了一套名为SciGuard的系统，用于控制科学AI模型的滥用风险。我们还提出了一套红队评估基准SciMT-Safety，以衡量不同系统的安全性。实验表明，所提出的SciGuard在良性测试中不牺牲性能的前提下，实现了最低的有害影响。最后，我们强调需要多学科协作努力，以确保科学领域AI模型的安全与伦理使用。希望本研究能在研究人员、从业者、政策制定者及公众中引发关于科学中伦理使用AI的富有成效的讨论，从而最大化利益，最小化滥用风险。