Since the introduction of Vision Transformer (ViT), patchification has long been regarded as a de facto image tokenization approach for plain visual architectures. By compressing the spatial size of images, this approach can effectively shorten the token sequence and reduce the computational cost of ViT-like plain architectures. In this work, we aim to thoroughly examine the information loss caused by this patchification-based compressive encoding paradigm and how it affects visual understanding. We conduct extensive patch size scaling experiments and excitedly observe an intriguing scaling law in patchification: the models can consistently benefit from decreased patch sizes and attain improved predictive performance, until it reaches the minimum patch size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable across different vision tasks, various input scales, and diverse architectures such as ViT and the recent Mamba models. Moreover, as a by-product, we discover that with smaller patches, task-specific decoder heads become less critical for dense prediction. In the experiments, we successfully scale up the visual sequence to an exceptional length of 50,176 tokens, achieving a competitive test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We hope this study can provide insights and theoretical foundations for future works of building non-compressive vision models. Code is available at https://github.com/wangf3014/Patch_Scaling.

翻译：自Vision Transformer (ViT) 提出以来，分块化一直被视作朴素视觉架构中图像标记化的默认方法。通过压缩图像的空间尺寸，该方法能有效缩短标记序列长度，降低类ViT朴素架构的计算成本。本研究旨在深入探究这种基于分块化的压缩编码范式所造成的信息损失，及其如何影响视觉理解。我们进行了广泛的分块尺寸缩放实验，并兴奋地观察到一个有趣的分块化缩放定律：模型能持续受益于减小的分块尺寸，并获得预测性能的提升，直至达到最小分块尺寸1x1，即像素级标记化。这一结论广泛适用于不同的视觉任务、多种输入尺度以及多样化的架构，如ViT和近期提出的Mamba模型。此外，作为附带发现，我们观察到当使用更小的分块时，针对特定任务的解码器头对于密集预测的重要性显著降低。在实验中，我们成功将视觉序列扩展至50,176个标记的惊人长度，在ImageNet-1k基准测试中，使用基础尺寸模型取得了84.6%的竞争性测试准确率。我们希望这项研究能为未来构建非压缩视觉模型的工作提供启示和理论基础。代码发布于 https://github.com/wangf3014/Patch_Scaling。