Transformer-based NLP models are powerful but have high computational costs that limit deployment. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and decomposable tasks where multiple outputs are required for a single shared input. Our method, prompt-in-decoder (PiD), encodes the input once and decodes the output in parallel, boosting both training and inference efficiency by avoiding duplicate input encoding and increasing the operational intensity (ratio of numbers of arithmetic operation to memory access) of decoding process by sharing the input key-value cache. We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks, with comparable or better performance.

翻译：基于Transformer的自然语言处理模型虽然功能强大，但存在计算成本高的问题，限制了其实际部署。在特定领域中，经过微调的编码器-解码器模型表现出色，其性能甚至可能超越GPT-4等规模更大、更通用的纯解码器模型。本文针对结构化输出和可分解任务（即单个共享输入需要生成多个输出的场景）提出了一种新型编码器-解码器配置方案，显著提升了计算效率。我们提出的"解码器中提示"（PiD）方法通过对输入进行单次编码并实现输出并行解码，避免了重复的输入编码过程，同时通过共享输入键值缓存提升了解码过程的运算强度（算术操作与内存访问次数之比）。该方法在训练和推理阶段均实现了效率提升，其计算量减少幅度大致与子任务数量成正比。在对话状态追踪、文本摘要和问答任务中，相比现有最优模型，我们的方法获得了最高4.6倍的加速效果，同时保持了相当或更优的性能表现。