Compression of large and performant vision foundation models (VFMs) into arbitrary bit-wise operations (BitOPs) allows their deployment on various hardware. We propose to fine-tune a VFM to a mixed-precision quantized supernet. The supernet-based neural architecture search (NAS) can be adopted for this purpose, which trains a supernet, and then subnets within arbitrary hardware budgets can be extracted. However, existing methods face difficulties in optimizing the mixed-precision search space and incurring large memory costs during training. To tackle these challenges, first, we study the effective search space design for fine-tuning a VFM by comparing different operators (such as resolution, feature size, width, depth, and bit-widths) in terms of performance and BitOPs reduction. Second, we propose memory-efficient supernet training using a low-rank adapter (LoRA) and a progressive training strategy. The proposed method is evaluated for the recently proposed VFM, Segment Anything Model, fine-tuned on segmentation tasks. The searched model yields about a 95% reduction in BitOPs without incurring performance degradation.

翻译：大型高性能视觉基础模型（VFM）压缩为任意比特运算（BitOPs）可在各类硬件上部署。我们提出将视觉基础模型微调为混合精度量化超网。为此可采用基于超网的神经架构搜索（NAS），通过训练超网，可提取出满足任意硬件预算的子网。然而现有方法在优化混合精度搜索空间时面临困难，且在训练过程中会产生较大的内存开销。为应对这些挑战，我们首先研究通过比较不同算子（如分辨率、特征尺寸、宽度、深度和位宽）在性能和比特运算缩减方面的效果，设计适用于视觉基础模型微调的高效搜索空间。其次，我们提出采用低秩适配器（LoRA）与渐进式训练策略的内存高效超网训练方法。所提方法在近期提出的视觉基础模型"分割一切模型"上进行了评估，针对分割任务进行微调。搜索得到的模型在性能不降级的情况下，比特运算量减少约95%。