Due to data privacy issues, accelerating networks with tiny training sets has become a critical need in practice. Previous methods achieved promising results empirically by filter-level pruning. In this paper, we both study this problem theoretically and propose an effective algorithm aligning well with our theoretical results. First, we propose the finetune convexity hypothesis to explain why recent few-shot compression algorithms do not suffer from overfitting problems. Based on it, a theory is further established to explain these methods for the first time. Compared to naively finetuning a pruned network, feature mimicking is proved to achieve a lower variance of parameters and hence enjoys easier optimization. With our theoretical conclusions, we claim dropping blocks is a fundamentally superior few-shot compression scheme in terms of more convex optimization and a higher acceleration ratio. To choose which blocks to drop, we propose a new metric, recoverability, to effectively measure the difficulty of recovering the compressed network. Finally, we propose an algorithm named PRACTISE to accelerate networks using only tiny training sets. PRACTISE outperforms previous methods by a significant margin. For 22% latency reduction, it surpasses previous methods by on average 7 percentage points on ImageNet-1k. It also works well under data-free or out-of-domain data settings. Our code is at https://github.com/DoctorKey/Practise

翻译：由于数据隐私问题，利用小训练集加速网络已成为实际应用中的关键需求。先前方法通过滤波器级剪枝在经验上取得了显著成果。本文从理论角度研究该问题，并提出与理论结果高度契合的有效算法。首先，我们提出微调凸性假设，以解释近期少样本压缩算法为何不会出现过拟合问题。基于此，首次建立了解释这些方法的理论框架。与对剪枝网络进行朴素微调相比，特征模拟被证明能实现更低的参数方差，从而更易于优化。基于理论结论，我们主张在更凸的优化空间和更高加速比方面，丢弃块结构本质上更优。为选择待丢弃的块，我们提出新指标"可恢复性"，以有效衡量压缩网络的恢复难度。最终提出名为PRACTISE的算法，仅使用小训练集实现网络加速。PRACTISE以显著优势超越先前方法：在延迟降低22%时，其在ImageNet-1k上的平均性能提升达7个百分点。该方法在无数据或域外数据场景下同样表现优异。我们的代码开源在https://github.com/DoctorKey/Practise