Running large language models locally is often impractical, pushing inference on sensitive text to third-party providers. Split inference partially mitigates this by keeping tokens on the client and sending only hidden representations, but these representations can still be recovered via nearest-neighbor search against the public embedding table. We propose an orthogonal obfuscation procedure in which the client multiplies embeddings by a secret orthogonal matrix before transmission. To enable correct inference under arbitrary rotations, we introduce ConjFormer, a transformer variant that is exactly $\mathrm{O}(d)$-equivariant via a lightweight normalization change (scalar RMSNorm) together with blockwise orthogonal conjugation of all linear weights. As a result, the server performs the full forward pass entirely in the rotated basis and never observes unrotated hidden states. Experiments on GPT-2 and Llama 3.2 1B models fine-tuned on PubMed show that orthogonal obfuscation eliminates direct cosine nearest-neighbor inversion and reduces token recovery from over 35% top-10 to at most 1.3%, while increasing perplexity by only 0.4% after fine-tuning. These results indicate that enforcing symmetry at the architectural level can provide a practical defense for privacy-preserving LLM inference without noise injection or heavy cryptographic machinery.

翻译：在本地运行大型语言模型通常不切实际，这导致对敏感文本的推理不得不交由第三方服务提供商完成。分割推理通过将词元保留在客户端，仅发送隐藏表示来部分缓解此问题，但这些表示仍可通过针对公共嵌入表的最近邻搜索被恢复。我们提出一种正交混淆机制，其中客户端在传输前使用秘密正交矩阵对嵌入进行乘法变换。为实现任意旋转下的正确推理，我们提出了ConjFormer——一种通过轻量级归一化改造（标量RMSNorm）结合所有线性权重的分块正交共轭而精确满足$\mathrm{O}(d)$-等变性的Transformer变体。因此，服务器完全在旋转后的基向量空间中执行完整前向传播，且始终无法观测到未旋转的隐藏状态。在针对PubMed微调的GPT-2与Llama 3.2 1B模型上的实验表明：正交混淆消除了直接的余弦最近邻反演，将词元恢复率从超过35%的top-10准确率降至至多1.3%，同时微调后困惑度仅增加0.4%。这些结果表明，在架构层面施加对称性可为隐私保护型大语言模型推理提供一种无需噪声注入或繁重密码学机制的实用防御手段。