Early exiting has become a promising approach to improving the inference efficiency of deep networks. By structuring models with multiple classifiers (exits), predictions for ``easy'' samples can be generated at earlier exits, negating the need for executing deeper layers. Current multi-exit networks typically implement linear classifiers at intermediate layers, compelling low-level features to encapsulate high-level semantics. This sub-optimal design invariably undermines the performance of later exits. In this paper, we propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task with a novel dual-branch architecture. A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks. Bi-directional cross-attention layers are established to progressively fuse the information of both branches. Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features. Dyn-Perceiver constitutes a versatile and adaptable framework that can be built upon various architectures. Experiments on image classification, action recognition, and object detection demonstrate that our method significantly improves the inference efficiency of different backbones, outperforming numerous competitive approaches across a broad range of computational budgets. Evaluation on both CPU and GPU platforms substantiate the superior practical efficiency of Dyn-Perceiver. Code is available at https://www.github.com/LeapLabTHU/Dynamic_Perceiver.

翻译：早期退出已成为提升深度网络推理效率的一种有前景的方法。通过使用多个分类器（出口）构建模型，可以在较早的出口处对“简单”样本生成预测，从而无需执行更深层的网络。当前的多出口网络通常在中间层实现线性分类器，迫使低层特征封装高层语义。这种次优设计不可避免地会损害后续出口的性能。本文提出动态感知器（Dyn-Perceiver），通过一种新颖的双分支架构解耦特征提取过程与早期分类任务。特征分支负责提取图像特征，而分类分支处理分配给分类任务的潜在编码。建立双向交叉注意力层，逐步融合两个分支的信息。早期出口仅放置在分类分支中，从而消除了低层特征需具有线性可分性的需求。Dyn-Perceiver构成一个多功能且可适应的框架，可基于多种架构构建。在图像分类、动作识别和对象检测上的实验表明，我们的方法显著提升了不同骨干网络的推理效率，在广泛的计算预算范围内优于众多竞争性方法。在CPU和GPU平台上的评估证实了Dyn-Perceiver优越的实际效率。代码可在https://www.github.com/LeapLabTHU/Dynamic_Perceiver获取。