The Languini Kitchen serves as both a research collective and codebase designed to empower researchers with limited computational resources to contribute meaningfully to the field of language modelling. We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. The number of tokens on which a model is trained is defined by the model's throughput and the chosen compute class. Notably, this approach avoids constraints on critical hyperparameters which affect total parameters or floating-point operations. For evaluation, we pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. On it, we compare methods based on their empirical scaling trends which are estimated through experiments at various levels of compute. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput. While the GPT baseline achieves better perplexity throughout all our levels of compute, our LSTM baseline exhibits a predictable and more favourable scaling law. This is due to the improved throughput and the need for fewer training tokens to achieve the same decrease in test perplexity. Extrapolating the scaling laws leads of both models results in an intersection at roughly 50,000 accelerator hours. We hope this work can serve as the foundation for meaningful and reproducible language modelling research.

翻译：Languini厨房既是一个研究集体，也是一个代码库，旨在赋能计算资源有限的研究人员为语言建模领域做出实质性贡献。我们提出了一套实验协议，能够基于等价计算量（以加速器小时数衡量）进行模型比较。模型训练的token数量由其吞吐量和所选计算类别共同决定。值得注意的是，该方法避免了对影响总参数量或浮点运算次数的关键超参数施加约束。在评估方面，我们对现有的大规模、多样且高质量的书籍数据集进行预处理，该数据集在质量、多样性和文档长度上均超越现有学术基准。基于该数据集，我们通过不同计算量级别的实验估算经验性缩放趋势，从而对比各类方法。本工作还提供了两个基线模型：一个源自GPT-2架构的前馈模型，以及一个以新型十倍吞吐量LSTM形式实现的循环模型。尽管GPT基线在我们所有计算量级别上均获得了更优的困惑度，但LSTM基线展现出可预测且更有利的缩放定律。这得益于其更高的吞吐量，以及实现相同测试困惑度降低所需的训练token更少。对两个模型缩放定律进行外推，结果在大约50,000加速器小时处存在交叉。我们希望本工作能为有意义且可复现的语言建模研究奠定基础。