Large Language Model (LLM) safety is one of the most pressing challenges for enabling wide-scale deployment. While most studies and global discussions focus on generic harms, such as models assisting users in harming themselves or others, enterprises face a more fundamental concern: whether LLM-based agents are safe for their intended use case. To address this, we introduce operational safety, defined as an LLM's ability to appropriately accept or refuse user queries when tasked with a specific purpose. We further propose OffTopicEval, an evaluation suite and benchmark for measuring operational safety both in general and within specific agentic use cases. Our evaluations on six model families comprising 20 open-weight LLMs reveal that while performance varies across models, all of them remain highly operationally unsafe. Even the strongest models - Qwen-3 (235B) with 77.77% and Mistral (24B) with 79.96% - fall far short of reliable operational safety, while GPT models plateau in the 62-73% range, Phi achieves only mid-level scores (48-70%), and Gemma and Llama-3 collapse to 39.53% and 23.84%, respectively. While operational safety is a core model alignment issue, to suppress these failures, we propose prompt-based steering methods: query grounding (Q-ground) and system-prompt grounding (P-ground), which substantially improve OOD refusal. Q-ground provides consistent gains of up to 23%, while P-ground delivers even larger boosts, raising Llama-3.3 (70B) by 41% and Qwen-3 (30B) by 27%. These results highlight both the urgent need for operational safety interventions and the promise of prompt-based steering as a first step toward more reliable LLM-based agents.

翻译：大型语言模型（LLM）的安全性是实现大规模部署最紧迫的挑战之一。尽管大多数研究和全球讨论都集中在通用危害上，例如模型协助用户伤害自己或他人，但企业面临一个更根本的担忧：基于LLM的智能体在其预期用例中是否安全。为解决此问题，我们引入了操作安全性，其定义为LLM在被赋予特定目的时，适当地接受或拒绝用户查询的能力。我们进一步提出了OffTopicEval，这是一个用于衡量通用及特定智能体用例中操作安全性的评估套件和基准。我们对包含20个开放权重LLM的六个模型系列进行的评估表明，尽管各模型性能存在差异，但它们均表现出高度的操作不安全。即使是最强的模型——Qwen-3（235B）达到77.77%，Mistral（24B）达到79.96%——也远未达到可靠的操作安全水平，而GPT模型在62-73%范围内趋于平稳，Phi仅获得中等分数（48-70%），Gemma和Llama-3则分别骤降至39.53%和23.84%。虽然操作安全性是模型对齐的核心问题，但为抑制这些失败，我们提出了基于提示的引导方法：查询接地（Q-ground）和系统提示接地（P-ground），这两种方法显著改善了分布外拒绝性能。Q-ground提供了高达23%的稳定增益，而P-ground带来了更大的提升，将Llama-3.3（70B）提高了41%，将Qwen-3（30B）提高了27%。这些结果既凸显了对操作安全性干预的迫切需求，也展现了基于提示的引导作为迈向更可靠基于LLM的智能体的第一步所具备的潜力。