Model parameter regularization is a widely used technique to improve generalization, but also can be used to shape the weight distributions for various purposes. In this work, we shed light on how weight regularization can assist model quantization and compression techniques, and then propose range regularization (R^2) to further boost the quality of model optimization by focusing on the outlier prevention. By effectively regulating the minimum and maximum weight values from a distribution, we mold the overall distribution into a tight shape so that model compression and quantization techniques can better utilize their limited numeric representation powers. We introduce L-inf regularization, its extension margin regularization and a new soft-min-max regularization to be used as a regularization loss during full-precision model training. Coupled with state-of-the-art quantization and compression techniques, models trained with R^2 perform better on an average, specifically at lower bit weights with 16x compression ratio. We also demonstrate that R^2 helps parameter constrained models like MobileNetV1 achieve significant improvement of around 8% for 2 bit quantization and 7% for 1 bit compression.

翻译：模型参数正则化是一种广泛用于提升泛化能力的技术，也可用于按特定需求塑造权重分布。本文揭示了权重正则化如何辅助模型量化与压缩技术，并提出范围正则化（R^2），通过聚焦异常值抑制进一步优化模型质量。通过有效调控权重分布的最小值和最大值，我们将整体分布塑造成紧凑形态，使模型压缩与量化技术能更充分利用其有限的数值表示能力。我们引入L-无穷正则化、其扩展的边界正则化以及新型软最小-最大正则化，作为全精度模型训练期间的正则化损失项。结合最先进的量化与压缩技术，基于R^2训练的模型在平均性能上表现更优，尤其在16倍压缩比的低位宽权重场景下。我们还证明，R^2可帮助MobileNetV1等参数受限模型实现显著提升：2比特量化性能提升约8%，1比特压缩性能提升约7%。