Large language models (LLMs) have a surprising failure: when trained on "A has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law - hence even if we train on the entire internet. This work proposes an alternative training scheme, called reverse training, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing the training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data-matched reverse-trained models provide superior performance to standard models on standard tasks, and compute-matched reverse-trained models provide far superior performance on reversal tasks, helping resolve the reversal curse issue.

翻译：大型语言模型（LLMs）存在一个惊人的缺陷：当训练数据中出现“A具有特征B”时，模型无法泛化到“B是A的特征”，这被称为逆转诅咒。即便模型使用数万亿个token进行训练，由于齐普夫定律的存在，此问题依然会出现——因此，即便在互联网的全部内容上进行训练也无法解决。本研究提出了一种替代训练方案，称为逆转训练，其中所有单词被使用两次，使可用token数量翻倍。LLM通过逆转训练字符串的同时保留（即不逆转）选定的子字符串（如实体），在正向和反向两个方向上进行训练。我们表明，在标准任务上，数据匹配的逆转训练模型优于标准模型；而在逆转任务上，计算匹配的逆转训练模型则提供远优于标准的性能，从而有助于解决逆转诅咒问题。