Gradient inversion attacks reveal that private training text can be reconstructed from shared gradients, posing a privacy risk to large language models (LLMs). While prior methods perform well in small-batch settings, scaling to larger batch sizes and longer sequences remains challenging due to severe signal mixing, high computational cost, and degraded fidelity. We present SOMP (Subspace-Guided Orthogonal Matching Pursuit), a scalable gradient inversion framework that casts text recovery from aggregated gradients as a sparse signal recovery problem. Our key insight is that aggregated transformer gradients retain exploitable head-wise geometric structure together with sample-level sparsity. SOMP leverages these properties to progressively narrow the search space and disentangle mixed signals without exhaustive search. Experiments across multiple LLM families, model scales, and five languages show that SOMP consistently outperforms prior methods in the aggregated-gradient regime.For long sequences at batch size B=16, SOMP achieves substantially higher reconstruction fidelity than strong baselines, while remaining computationally competitive. Even under extreme aggregation (up to B=128), SOMP still recovers meaningful text, suggesting that privacy leakage can persist in regimes where prior attacks become much less effective.

翻译：梯度反演攻击表明，共享的梯度可能泄露私有训练文本，这对大语言模型（LLMs）构成了隐私风险。现有方法在小批量场景下表现良好，但由于严重的信号混叠、高昂的计算成本以及重建保真度下降，将其扩展至更大批量规模和更长序列仍具挑战性。本文提出SOMP（子空间引导正交匹配追踪），一种可扩展的梯度反演框架，将聚合梯度中的文本恢复问题转化为稀疏信号恢复问题。我们的核心洞见是：聚合后的Transformer梯度保留了可利用的头部几何结构以及样本级稀疏性。SOMP利用这些特性逐步缩小搜索空间，并在无需穷举搜索的情况下解耦混叠信号。在多种LLM架构、模型规模及五种语言上的实验表明，在聚合梯度场景下，SOMP始终优于现有方法。对于批量大小B=16的长序列，SOMP在保持计算效率竞争力的同时，实现了显著高于基线方法的重建保真度。即使在极端聚合条件下（B高达128），SOMP仍能恢复有意义的文本，这表明在现有攻击方法效果大幅下降的场景中，隐私泄露风险可能依然存在。