Language models have seen significant growth in the size of their corpus, leading to notable performance improvements. Yet, there has been limited progress in developing models that handle smaller, more human-like datasets. As part of the BabyLM shared task, this study explores the impact of reinforcement learning from human feedback (RLHF) on language models pretrained from scratch with a limited training corpus. Comparing two GPT-2 variants, the larger model performs better in storytelling tasks after RLHF fine-tuning. These findings suggest that RLHF techniques may be more advantageous for larger models due to their higher learning and adaptation capacity, though more experiments are needed to confirm this finding. These insights highlight the potential benefits of RLHF fine-tuning for language models within limited data, enhancing their ability to maintain narrative focus and coherence while adhering better to initial instructions in storytelling tasks. The code for this work is publicly at https://github.com/Zephyr1022/BabyStories-UTSA.

翻译：语言模型的语料库规模显著增长，带来了性能的明显提升。然而，在开发处理更小、更接近人类数据集的模型方面进展有限。作为BabyLM共享任务的一部分，本研究探讨了从人类反馈中进行的强化学习（RLHF）对从头开始使用有限训练语料库预训练的语言模型的影响。通过比较两个GPT-2变体，经过RLHF微调后，较大的模型在讲故事任务中表现更优。这些发现表明，RLHF技术可能对较大模型更具优势，因其拥有更高的学习与适应能力，但尚需更多实验以确认这一结论。这些见解凸显了RLHF微调在有限数据下对语言模型的潜在益处——在讲故事任务中，它能增强模型维持叙事焦点与连贯性的能力，同时更好地遵循初始指令。本工作的代码已公开于 https://github.com/Zephyr1022/BabyStories-UTSA。