Large Language Models have many methods for solving the same problem. This introduces novel strengths (different methods may work well for different problems) and weaknesses (it may be difficult for users to know which method to use). In this paper, we introduce Multi-Method Self-Training (MMST), where one method is trained on the filtered outputs of another, allowing us to augment the strengths and ameliorate the weaknesses of each method. Using a 176B parameter model trained on both language and code, we show that MMST can 1) improve the less performant method (up to 30%) making the model easier to use, 2) improve the more performant method (up to 32.2%) making the model more performant, and 3) improve the performance of related but distinct tasks (up to 10.3%) by improving the ability of the model to generate rationales. We then conduct ablation analyses to explore why MMST works. We show that MMST generates more data than traditional self-training, but the improvement in performance is driven by the use of multiple methods. We also analyze prompt-engineering and anti-correlated performance between methods as means of making MMST more effective. We hope the evidence from our paper motivates machine learning researchers to explore ways in which advances in language models allow for new forms of training.

翻译：大语言模型在解决同一问题时拥有多种方法。这带来了新的优势（不同方法可能适用于不同问题）和劣势（用户可能难以确定使用哪种方法）。本文提出多方法自我训练（Multi-Method Self-Training, MMST），通过对一种方法过滤后的输出结果训练另一种方法，从而增强各方法的优势并弥补其不足。基于一个同时训练于语言和代码的176B参数模型，我们证明MMST能够：1）提升性能较弱的模型方法（最高达30%），使模型更易使用；2）提升性能较强的模型方法（最高达32.2%），使模型性能更优；3）通过增强模型生成推理依据的能力，提升相关但不同任务的性能（最高达10.3%）。随后我们通过消融分析探究MMST的有效性机制，发现MMST比传统自我训练生成更多数据，但性能提升的核心驱动力在于多方法的协同使用。我们还分析了提示工程与各方法间负相关性能对提升MMST效果的潜在作用。希望本文提供的证据能激励机器学习研究者探索如何利用语言模型的进步创造新型训练范式。