Equivariance is an important feature in machine learning, including language models. It ensures that any sequences of phrases with the same meanings are interpreted consistently. For example, the sentence 'There is a cat on the table' should be interpreted by language models as it is, regardless of variations in its token-level expression. Building on this insight, I propose a new theory suggesting that insufficient equivariance in language models can lead to hallucinations. According to this theory, which is both intuitive and novel, language models trained on relatively small datasets tend to misinterpret input texts and/or generate incorrect texts (i.e., hallucinations). To test this theory, I developed a toy model known as 'dancing men', which is a character-level substitution cipher. Additionally, I propose a novel technique based on the T5 (Text To Text Transfer Transformer) model to efficiently decipher these codes without relying on frequency analysis. I have found that this T5 model can almost completely solve the cipher, demonstrating its ability to acquire equivariance in this frame. This method could be scaled up to word-level and sentence-level substitution ciphers, analogous to large language models without tokenizers or dictionaries. This scalability makes it suitable for investigating the proposed link between inadequate equivariance acquisition and the emergence of hallucinations.

翻译：等变性是机器学习（包括语言模型）中的一个重要特性。它确保具有相同含义的任意短语序列能够被一致地解释。例如，句子“桌上有一只猫”应被语言模型按其原意理解，无论其在词元层面的表达形式如何变化。基于这一见解，我提出一个新的理论：语言模型中等变性不足可能导致幻觉。根据这一既直观又新颖的理论，在相对较小数据集上训练的语言模型倾向于误解输入文本和/或生成错误文本（即幻觉）。为验证该理论，我开发了一个名为“跳舞小人”的玩具模型，它实质上是一种字符级替换密码。此外，我提出了一种基于T5（Text To Text Transfer Transformer，文本到文本转换Transformer）模型的新技术，能够在无需频率分析的情况下高效破译这些密码。我发现该T5模型几乎能完全破解该密码，证明了它能在该框架中获取等变性。此方法可扩展至词级和句子级替换密码，类似于无分词器或无字典的大型语言模型。这种可扩展性使其非常适合研究所提出的不充分等变性获取与幻觉产生之间的关联。