The widespread deployment of large language models (LLMs) has raised growing concerns about their misuse risks and associated safety issues. While prior studies have examined the safety of LLMs in general usage, code generation, and agent-based applications, their vulnerabilities in automated algorithm design remain underexplored. To fill this gap, this study investigates this overlooked safety vulnerability, with a particular focus on intelligent optimization algorithm design, given its prevalent use in complex decision-making scenarios. We introduce MalOptBench, a benchmark consisting of 60 malicious optimization algorithm requests, and propose MOBjailbreak, a jailbreak method tailored for this scenario. Through extensive evaluation of 13 mainstream LLMs including the latest GPT-5 and DeepSeek-V3.1, we reveal that most models remain highly susceptible to such attacks, with an average attack success rate of 83.59% and an average harmfulness score of 4.28 out of 5 on original harmful prompts, and near-complete failure under MOBjailbreak. Furthermore, we assess state-of-the-art plug-and-play defenses that can be applied to closed-source models, and find that they are only marginally effective against MOBjailbreak and prone to exaggerated safety behaviors. These findings highlight the urgent need for stronger alignment techniques to safeguard LLMs against misuse in algorithm design.

翻译：大型语言模型（LLMs）的广泛部署引发了对其滥用风险及相关安全问题的日益关注。尽管先前研究已考察了LLMs在通用使用、代码生成和基于智能体的应用中的安全性，但其在自动化算法设计中的脆弱性仍未得到充分探究。为填补这一空白，本研究聚焦于这一被忽视的安全漏洞，特别关注智能优化算法设计领域，鉴于其在复杂决策场景中的普遍应用。我们引入了MalOptBench，一个包含60个恶意优化算法请求的基准测试集，并提出了针对此场景定制的越狱方法MOBjailbreak。通过对包括最新GPT-5和DeepSeek-V3.1在内的13个主流LLMs进行广泛评估，我们发现大多数模型对此类攻击仍高度易感：在原始恶意提示下，平均攻击成功率达83.59%，平均有害性评分（5分制）为4.28；而在MOBjailbreak攻击下，模型防御近乎完全失效。此外，我们评估了可应用于闭源模型的最先进即插即用防御方法，发现它们对MOBjailbreak仅具有微弱效果，且容易产生过度安全行为。这些发现凸显了亟需更强的对齐技术来保护LLMs，防止其在算法设计中被滥用。