Recent achievements of vision-language models in end-to-end OCR point to a new avenue for low-loss compression of textual information. This motivates earlier works that render the Transformer's input into images for prefilling, which effectively reduces the number of tokens through visual encoding, thereby alleviating the quadratically increased Attention computations. However, this partial compression fails to save computational or memory costs at token-by-token inference. In this paper, we investigate global context compression, which saves tokens at both prefilling and inference stages. Consequently, we propose VIST2, a novel Transformer that interleaves input text chunks alongside their visual encoding, while depending exclusively on visual tokens in the pre-context to predict the next text token distribution. Around this idea, we render text chunks into sketch images and train VIST2 in multiple stages, starting from curriculum-scheduled pretraining for optical language modeling, followed by modal-interleaved instruction tuning. We conduct extensive experiments using VIST2 families scaled from 0.6B to 8B to explore the training recipe and hyperparameters. With a 4$\times$ compression ratio, the resulting models demonstrate significant superiority over baselines on long writing tasks, achieving, on average, a 3$\times$ speedup in first-token generation, 77% reduction in memory usage, and 74% reduction in FLOPS. Our codes and datasets will be public to support further studies.

翻译：视觉语言模型在端到端OCR领域的最新成就，为文本信息的低损失压缩开辟了新途径。这推动了早期将Transformer输入渲染为图像进行预填充的研究，该方法通过视觉编码有效减少token数量，从而缓解注意力计算量的二次增长。然而，这种局部压缩无法在逐token推理阶段节省计算或内存成本。本文研究全局上下文压缩技术，实现在预填充和推理阶段同时减少token消耗。为此，我们提出VIST2——一种创新的Transformer架构，它将输入文本块与其视觉编码交错排列，并仅依赖前文语境中的视觉token来预测下一个文本token的分布。基于此设计，我们将文本块渲染为草图图像，并通过多阶段训练VIST2：从课程调度的光学语言建模预训练开始，随后进行模态交错指令微调。我们使用参数量从0.6B到8B的VIST2系列模型开展大量实验，以探索训练方案和超参数设置。在4倍压缩比条件下，所得模型在长文本生成任务中显著优于基线，平均实现首token生成速度提升3倍、内存使用降低77%、浮点运算量减少74%的优化效果。我们的代码与数据集将公开以支持后续研究。