FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals. We identify a counterintuitive source of this failure: dominant activation outliers are not merely arbitrary sparse events, but are largely induced by a coherent rank-one mean bias, whose direction aligns with the leading anisotropic spectral component. This mean component strengthens during training, is amplified and reshaped by attention and FFN operators, and increasingly dominates top activation magnitudes. Crucially, this discovery reveals that a seemingly complex outlier-suppression problem admits a truly simple solution: isolate the coherent mean before quantization. We therefore propose Averis, a mean-residual splitting quantization method that separates the mean component using only reductions and elementwise subtractions before FP4 quantization. Across Qwen3 0.6B Dense trained on 100B tokens and Qwen3 7B A1.5B MoE trained on 50B tokens, Averis enables robust W4A4G4 FP4 training, reducing BF16 loss gaps to 1.19%/0.81% versus 2.05%/1.10% for NVIDIA's recently released Hadamard-based outlier-smoothing method, while limiting downstream gaps to 0.89/0.71 points. With only 2.20% end-to-end overhead over vanilla NVFP4, about 30% of NVIDIA's Hadamard-based design, Averis provides a hardware-efficient path to stable low-bit LLM training. Complementary to Hadamard, Averis further reduces the Qwen3-0.6B loss and downstream gaps to 0.94% and 0.73 points when combined. Code is available at: https://anonymous.4open.science/r/averis-504D.

翻译：FP4训练有望为大型语言模型带来显著的内存与计算节省，但由于逐块量化受极端激活值主导——这些激活值扩大动态范围并压缩长尾信号——导致其稳定性脆弱。我们识别出这一失效的反直觉根源：主导离群值并非单纯的稀疏偶发事件，而是主要由相干秩一均值偏差诱导产生，该偏差方向与主导各向异性谱分量对齐。该均值分量在训练过程中持续增强，经注意力与前馈神经网络操作放大重塑，并日益主导顶层激活幅值。关键突破在于：这一发现表明，看似复杂的离群抑制问题存在一个极简解法——在量化前分离相干均值。为此我们提出Averis方法，这是一种均值残差分离量化技术，仅通过归约运算与逐元素减法即可在FP4量化前分离均值分量。在基于100B tokens训练的Qwen3 0.6B密集模型与基于50B tokens训练的Qwen3 7B A1.5B MoE模型上，Averis实现了稳健的W4A4G4 FP4训练，将BF16损失差距分别降至1.19%/0.81%（对比NVIDIA近期发布的基于Hadamard的离群平滑方法的2.05%/1.10%），同时将下游指标差距限制在0.89/0.71个点。相比原始NVFP4仅引入2.20%端到端开销（约为NVIDIA Hadamard设计方案的30%），Averis为低比特LLM稳定训练提供了硬件高效路径。作为Hadamard方法的补充，Averis在联合使用时进一步将Qwen3-0.6B的损失与下游指标差距降至0.94%与0.73个点。代码开源地址：https://anonymous.4open.science/r/averis-504D。