We present a systematic empirical study of transformer compression through over 40 experiments on GPT-2 (124M parameters) and Mistral 7B (7.24B parameters). Our analysis covers spectral compression, block-level function replacement, rotation-based quantization, activation geometry, and adaptive early exit. We identify five structural properties relevant to compression. (1) Variance is not importance: high-variance activation directions are approximately 96 percent uncorrelated with predictive directions (measured via CCA), and projecting onto these subspaces preserves over 90 percent of variance while degrading perplexity. (2) Block linearity is conditional: transformer blocks are approximately linear (R^2 ~ 0.95 on GPT-2, 0.93 on Mistral block 31) only under the correct upstream distribution; modifying earlier blocks induces distribution shift that degrades downstream approximations. (3) The reconstruction wall: approaches that factor weights into quantized components amplify errors through cross-terms, making direct quantization strictly superior. (4) Linearity increases with depth: Mistral 7B exhibits a progression from R^2 = 0.17 (block 0) to R^2 = 0.93 (block 31), indicating a division between nonlinear feature construction and linear refinement. (5) Approximately 30 percent of tokens are computationally easy, confirmed via exit heads and KL divergence sensitivity. We demonstrate that single-block linear replacement achieves 34x compression with a 1.71 perplexity increase on the final block of Mistral 7B, while multi-block replacement fails due to residual error accumulation and distribution shift. These findings suggest fundamental limits to static post-training compression and motivate adaptive, per-token computation as a more effective direction.

翻译：我们通过40余项实验，对GPT-2（1.24亿参数）与Mistral 7B（72.4亿参数）展开系统性实证研究，分析涵盖谱压缩、模块级功能替换、基于旋转的量化、激活几何结构及自适应早期退出机制。研究识别出与压缩相关的五项结构特性：（1）方差并非重要性：高方差激活方向与预测方向（通过CCA测量）的无关性高达约96%，投影至这些子空间可保留90%以上方差却导致困惑度恶化；（2）模块线性具有条件性：仅当上游分布正确时，Transformer模块呈现近似线性（GPT-2 R²≈0.95，Mistral第31层R²≈0.93），前置模块修改引发的分布偏移会降低下游近似质量；（3）重建困境：将权重分解为量化分量的方法会通过交叉项放大误差，使得直接量化更具优越性；（4）线性随深度递增：Mistral 7B从第0层R²=0.17增至第31层R²=0.93，表明存在非线性特征构建与线性精调的分界；（5）约30%的token计算难度较低，该结论经退出头与KL散度敏感性验证。我们证明对Mistral 7B最终层实施单模块线性替换可实现34倍压缩且困惑度仅增加1.71，而多模块替换因残差累积与分布偏移而失效。这些发现揭示了静态训练后压缩的固有局限性，并推动自适应逐token计算成为更有效的研究方向。