Using the bit string generation problem as a case study, we theoretically compare two standard methods for adapting large language models to new tasks. The first, referred to as supervised fine-tuning, involves training a new next token predictor on good generations. The second method, Best-of-N, trains a reward model to select good responses from a collection generated by an unaltered base model. If the learning setting is realizable, we find that supervised fine-tuning outperforms BoN through a better dependence on the response length in its rate of convergence. If realizability fails, then depending on the failure mode, BoN can enjoy a better rate of convergence in either n or a rate of convergence with better dependence on the response length.

翻译：以比特串生成问题为案例研究，我们从理论上比较了将大型语言模型适配到新任务的两种标准方法。第一种方法称为监督微调，即对优质生成结果训练新的下一个词元预测器。第二种方法Best-of-N则训练奖励模型，从原始基座模型生成的候选集中选择优质响应。研究发现，若学习场景是可实现的，监督微调在收敛速度上对响应长度的依赖性更优，从而优于BoN。若可实现性不成立，则根据失效模式的不同，BoN可能在n维度上获得更优收敛速度，或是在响应长度依赖性上实现更优收敛速率。