Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning. The fine-tuning data is generally sequentially concatenated from a specific task instruction, an input sentence, and the corresponding response. Considering the locality modeled by the self-attention mechanism of LLMs, these models face the risk of instruction forgetting when generating responses for long input sentences. To mitigate this issue, we propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences. Theoretical analysis suggests that our straightforward method can alter the model's learning focus, thereby emphasizing the training of instruction-following capabilities. Concurrently, experimental results demonstrate that our approach consistently outperforms traditional settings across various model scales (1B / 7B / 13B) and different sequence generation tasks (translation and summarization), without any additional data or annotation costs. Notably, our method significantly improves the zero-shot performance on conditional sequence generation, e.g., up to 9.7 BLEU points on WMT zero-shot translation tasks.

翻译：大语言模型（LLMs）能够通过指令微调执行条件序列生成任务，例如翻译或摘要生成。微调数据通常由特定任务指令、输入句子和对应响应顺序拼接而成。考虑到LLMs的自注意力机制对局部性的建模，这些模型在生成长输入句子的响应时面临指令遗忘的风险。为缓解这一问题，我们提出通过将任务指令的位置移至输入句子之后来增强LLMs的指令遵循能力。理论分析表明，这一直接方法能够改变模型的学习焦点，从而强调指令遵循能力的训练。同时，实验结果表明，我们的方法在不同模型规模（1B/7B/13B）和不同序列生成任务（翻译与摘要）中始终优于传统设置，且无需额外数据或标注成本。值得注意的是，我们的方法显著提升了条件序列生成的零样本性能，例如在WMT零样本翻译任务上最高提升9.7个BLEU分。