Language models read long contexts and score well on knowledge and reasoning benchmarks, but neither shows whether a model can do work whose next step depends on everything that came before, as a long-running agent must. We call this ability task modeling and propose counting identical items as its simplest test: every item adds to the total and no item records it. We define counting capacity (CC) as the sequence length up to which a model returns the exact count more often than not, and measure it for 205 models. Every resolved measurement loses count abruptly, at a median of 50 items and at most 627; across release years the median rose 1.6-fold while context windows grew about 80-fold. Changing the counted symbol or the delimiter moves a model's CC severalfold, yet the model accounts for about three quarters of the variance across 16 models and 11 encodings. Reasoning leaves the median CC unchanged across 77 matched pairs. In six open-weight models the count stays readable in early layers past the boundary but does not reach the output, and an activation edit that carries the count forward also carries the state of an object-tracking task. CC tracks the METR task horizon after controlling for general capability, and rises with capacity on grid-execution tasks and, with reasoning off, on object tracking. We do not show that counting is necessary or sufficient for long-horizon work; that remains a hypothesis, which this cheap measurement can test.
翻译:暂无翻译