The state of the arts in vision-language pretraining (VLP) achieves exemplary performance but suffers from high training costs resulting from slow convergence and long training time, especially on large-scale web datasets. An essential obstacle to training efficiency lies in the entangled prediction rate (percentage of tokens for reconstruction) and corruption rate (percentage of corrupted tokens) in masked language modeling (MLM), that is, a proper corruption rate is achieved at the cost of a large portion of output tokens being excluded from prediction loss. To accelerate the convergence of VLP, we propose a new pretraining task, namely, free language modeling (FLM), that enables a 100% prediction rate with arbitrary corruption rates. FLM successfully frees the prediction rate from the tie-up with the corruption rate while allowing the corruption spans to be customized for each token to be predicted. FLM-trained models are encouraged to learn better and faster given the same GPU time by exploiting bidirectional contexts more flexibly. Extensive experiments show FLM could achieve an impressive 2.5x pretraining time reduction in comparison to the MLM-based methods, while keeping competitive performance on both vision-language understanding and generation tasks. Code will be public at https://github.com/TencentARC/FLM.

翻译：当前最先进的视觉语言预训练（VLP）方法虽取得了优异性能，但面临收敛缓慢、训练时间长导致的高昂训练成本问题，尤其是在大规模网络数据集上。训练效率的核心障碍在于掩码语言建模（MLM）中纠缠的预测率（用于重建的令牌比例）与损坏率（被损坏令牌比例），即需要以大量输出令牌被排除在预测损失之外为代价，才能获得合适的损坏率。为加速VLP收敛，我们提出一种新型预训练任务——自由语言建模（FLM），该任务可在任意损坏率下实现100%的预测率。FLM成功将预测率从与损坏率的绑定中解放出来，同时允许为每个待预测令牌定制损坏跨度。在相同GPU时间内，FLM训练的模型通过更灵活地利用双向上下文，能够学得更快更好。大量实验表明，与基于MLM的方法相比，FLM可实现2.5倍的预训练时间缩减，同时在视觉语言理解与生成任务上保持竞争性能。代码将在https://github.com/TencentARC/FLM 公开。