Speculative decoding (SD) has proven effective for accelerating LLM inference by quickly generating draft tokens and verifying them in parallel. However, SD remains largely unexplored for Large Vision-Language Models (LVLMs), which extend LLMs to process both image and text prompts. To address this gap, we benchmark existing inference methods with small draft models on 11 datasets across diverse input scenarios and observe scenario-specific performance fluctuations. Motivated by these findings, we propose Test-time Adaptive Batched Ensemble Drafting (TABED), which dynamically ensembles multiple drafts obtained via batch inference by leveraging deviations from past ground truths available in the SD setting. The dynamic ensemble method achieves an average robust walltime speedup of 1.74x over autoregressive decoding and a 5% improvement over single drafting methods, while remaining training-free and keeping ensembling costs negligible through parameter sharing. With its plug-and-play compatibility, we further enhance TABED by integrating advanced verification and alternative drafting methods. Code and custom-trained models are available at https://github.com/furiosa-ai/TABED.

翻译：推测解码（SD）通过快速生成草稿令牌并进行并行验证，已被证明能有效加速大语言模型（LLM）的推理过程。然而，对于将LLM扩展至能同时处理图像和文本输入的大型视觉语言模型（LVLMs）而言，推测解码的应用仍鲜有探索。为填补这一空白，我们在涵盖多样化输入场景的11个数据集上，使用小型草稿模型对现有推理方法进行了基准测试，并观察到性能随场景波动的现象。基于此发现，我们提出了测试时自适应批量集成草稿生成方法（TABED），该方法利用SD场景中可获取的历史真实值偏差，对通过批量推理获得的多个草稿进行动态集成。这一动态集成方法相比自回归解码实现了平均1.74倍的鲁棒墙上时间加速，较单一草稿生成方法提升5%的性能，同时保持无需训练的特性，并通过参数共享使集成成本可忽略不计。凭借其即插即用的兼容性，我们进一步通过集成高级验证与替代草稿生成方法增强了TABED。代码及定制训练模型发布于 https://github.com/furiosa-ai/TABED。