Language models (LMs) can express factual knowledge involving numeric properties such as Karl Popper was born in 1902. However, how this information is encoded in the model's internal representations is not understood well. Here, we introduce a simple method for finding and editing representations of numeric properties such as an entity's birth year. Empirically, we find low-dimensional subspaces that encode numeric properties monotonically, in an interpretable and editable fashion. When editing representations along directions in these subspaces, LM output changes accordingly. For example, by patching activations along a "birthyear" direction we can make the LM express an increasingly late birthyear: Karl Popper was born in 1929, Karl Popper was born in 1957, Karl Popper was born in 1968. Property-encoding directions exist across several numeric properties in all models under consideration, suggesting the possibility that monotonic representation of numeric properties consistently emerges during LM pretraining. Code: https://github.com/bheinzerling/numeric-property-repr

翻译：语言模型（LMs）能够表达涉及数值属性的事实知识，例如卡尔·波普尔出生于1902年。然而，这类信息如何在模型内部表示中被编码尚不明确。本文提出了一种简单方法，用于发现和编辑数值属性（如实体的出生年份）的表示。实验表明，我们发现了以单调方式编码数值属性的低维子空间，这些子空间具有可解释性和可编辑性。当沿着这些子空间的方向编辑表示时，LM的输出会相应改变。例如，通过沿“出生年份”方向修补激活值，可使LM表达逐渐增大的出生年份：卡尔·波普尔出生于1929年，卡尔·波普尔出生于1957年，卡尔·波普尔出生于1968年。在所考察的所有模型中，多个数值属性均存在属性编码方向，这表明数值属性的单调表示可能在LM预训练过程中一致涌现。代码：https://github.com/bheinzerling/numeric-property-repr