Understanding the latent space geometry of large language models (LLMs) is key to interpreting their behavior and improving alignment. Yet it remains unclear to what extent LLMs linearly organize representations related to semantic understanding. To explore this, we conduct a large-scale empirical study of hidden representations in 11 autoregressive models across six scientific topics. We find that high-level semantic information consistently resides in low-dimensional subspaces that form linearly separable representations across domains. This separability becomes more pronounced in deeper layers and under prompts that elicit structured reasoning or alignment behavior$\unicode{x2013}$even when surface content remains unchanged. These findings motivate geometry-aware tools that operate directly in latent space to detect and mitigate harmful and adversarial content. As a proof of concept, we train an MLP probe on final-layer hidden states as a lightweight latent-space guardrail. This approach substantially improves refusal rates on malicious queries and prompt injections that bypass both the model's built-in safety alignment and external token-level filters.

翻译：理解大语言模型（LLMs）的潜在空间几何结构是解释其行为和改进对齐的关键。然而，目前尚不清楚LLMs在多大程度上线性地组织与语义理解相关的表示。为探究此问题，我们对11个自回归模型在六个科学主题上的隐藏表示进行了大规模实证研究。我们发现，高层语义信息始终存在于低维子空间中，这些子空间在不同领域内形成线性可分的表示。这种可分性在更深层网络以及激发结构化推理或对齐行为的提示下变得更加明显——即使表面内容保持不变。这些发现启发了直接在潜在空间中运行的几何感知工具，用于检测和缓解有害及对抗性内容。作为概念验证，我们在最终层隐藏状态上训练了一个MLP探针，作为轻量级的潜在空间防护栏。该方法显著提高了对恶意查询和提示注入的拒绝率，这些查询和注入能够绕过模型内置的安全对齐机制和外部令牌级过滤器。