Large language models (LLMs) can perform impressive feats with in-context learning or lightweight finetuning. It is natural to wonder how well these models adapt to genuinely new tasks, but how does one find tasks that are unseen in internet-scale training sets? We turn to a field that is explicitly motivated and bottlenecked by a scarcity of web data: low-resource languages. In this paper, we introduce MTOB (Machine Translation from One Book), a benchmark for learning to translate between English and Kalamang -- a language with less than 200 speakers and therefore virtually no presence on the web -- using several hundred pages of field linguistics reference materials. This task framing is novel in that it asks a model to learn a language from a single human-readable book of grammar explanations, rather than a large mined corpus of in-domain data, more akin to L2 learning than L1 acquisition. We demonstrate that baselines using current LLMs are promising but fall short of human performance, achieving 44.7 chrF on Kalamang to English translation and 45.8 chrF on English to Kalamang translation, compared to 51.6 and 57.0 chrF by a human who learned Kalamang from the same reference materials. We hope that MTOB will help measure LLM capabilities along a new dimension, and that the methods developed to solve it could help expand access to language technology for underserved communities by leveraging qualitatively different kinds of data than traditional machine translation.

翻译：大型语言模型（LLMs）可通过上下文学习或轻量级微调展现令人印象深刻的能力。人们自然好奇这些模型如何适应真正的新任务，但如何找到互联网规模训练集中未见过的任务？我们转向一个因网络数据稀缺而受到明确动机和瓶颈限制的领域：低资源语言。本文提出MTOB（从一本书中学习机器翻译）基准测试，旨在通过数百页实地语言学参考资料，学习英语与卡拉芒语（一种仅有不到200人使用、因此在网络上几乎不存在的语言）之间的翻译。这种任务框架的新颖之处在于，它要求模型从一本可读的语法解释书中习得一种语言，而非从大量语料库中挖掘领域内数据，更接近第二语言习得而非母语习得。我们证明，基于当前LLMs的基线方法表现前景良好但仍不及人类水平：在卡拉芒语至英语翻译中达到44.7 chrF，英语至卡拉芒语翻译中达到45.8 chrF，而使用相同参考资料学习卡拉芒语的人类分别达到51.6和57.0 chrF。我们希望MTOB能帮助沿新维度衡量LLMs能力，且解决该问题所开发的方法可通过利用与传统机器翻译性质不同的数据，为资源匮乏的社群扩展语言技术可及性。