We introduce Bitune, a method that improves instruction-tuning of pretrained decoder-only large language models, leading to consistent gains on downstream tasks. Bitune applies both causal and bidirectional attention to the prompt, to obtain a better representation of the query or instruction. We realize this by introducing two sets of parameters, for which we apply parameter-efficient finetuning techniques. These causal and bidirectional features are then combined into a weighted average with trainable coefficients, which is subsequently used to generate new tokens. We demonstrate significant improvements in zero-shot performance on commonsense reasoning, arithmetic, and language understanding tasks, while extensive ablation studies validate the role of each component and demonstrate the method's agnosticism to different PEFT techniques.

翻译：本文提出Bitune方法，该方法通过改进预训练仅解码器大语言模型的指令微调过程，在下游任务中实现持续性能提升。Bitune对提示词同时应用因果注意力与双向注意力机制，以获得查询或指令的更优表征。我们通过引入两组参数实现这一目标，并对其应用参数高效微调技术。随后，这些因果与双向特征通过可训练系数加权平均进行融合，并用于生成新词元。实验证明，该方法在常识推理、算术运算及语言理解任务的零样本性能上取得显著提升，同时大量消融研究验证了各组件的贡献，并证明该方法对不同参数高效微调技术具有普适性。