Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rotations, Hadamard transform, channel-wise scaling) have been successfully applied to reduce post-training quantization error, yet a principled explanation remains elusive. We analyze linear-layer quantization via the signal-to-quantization-noise ratio (SQNR), showing that for uniform integer quantization at a fixed bit width, SQNR decomposes into (i) the concentration of weights and activations (capturing spread and outliers), and (ii) the alignment of their dominant variation directions. This reveals an actionable insight: beyond concentration - the focus of most prior transforms (e.g. rotations or Hadamard) - improving alignment between weight and activation can further reduce quantization error. Motivated by this, we introduce block Concentration-Alignment Transforms (CAT), a lightweight linear transformation that uses a covariance estimate from a small calibration set to jointly improve concentration and alignment, approximately maximizing SQNR. Experiments across several LLMs show that CAT consistently matches or outperforms prior transform-based quantization methods at 4-bit precision, confirming the insights gained in our framework.

翻译：量化能显著提升大型语言与视觉模型的效率，但通常会导致精度下降。近期，函数保持变换（如旋转、哈达玛变换、通道级缩放）已成功应用于降低训练后量化误差，但其原理性解释仍不明确。我们通过信噪比（SQNR）分析线性层量化，表明在固定位宽下采用均匀整数量化时，SQNR可分解为：（i）权重与激活的集中度（反映分布范围与异常值），以及（ii）二者主导变化方向的对齐性。这揭示了一个可操作的见解：除集中度——多数现有变换（如旋转或哈达玛变换）的关注焦点外——提升权重与激活间的对齐性能进一步降低量化误差。受此启发，我们提出块集中对齐变换（CAT），这是一种轻量级线性变换方法，利用小型校准集估计的协方差信息联合优化集中度与对齐性，从而近似最大化SQNR。在多个大语言模型上的实验表明，在4比特精度下，CAT始终匹配或优于现有基于变换的量化方法，验证了我们理论框架所获得的洞见。