Many current paths to more advanced AI depend on the assumption that large language models (LLMs) can generalize learned relationships to solve complex, out-of-distribution (OOD) problems. However, this is not an easy quality to measure. For most real-world problems and benchmarks it suffices to exploit a few memorized subproblems or a small fraction of the available data to produce a correct solution. This is interpolation. Generalization requires the capacity to make use of all available data to solve problems without these shortcuts. To test this quality, we introduce a synthetic Boolean circuit reconstruction benchmark over $GF(2)$. We use an adversarial sampling oracle to block partial-information shortcuts, ensuring that each step requires the integration of all historical context with new evidence. Our evaluations reveal that standalone LLMs fall significantly short of sustained reasoning: as problem depth increases, both small and frontier models exhibit a catastrophic collapse in their ability to accurately predict the next logical step. However, we also demonstrate that tool synthesis provides a remedy to this depth-induced reasoning collapse. When allowed to generate, execute, and iteratively refine code, even small architectures can sustain accurate reasoning over long horizons and rival frontier model performance on this task. These results indicate that synthesizing tools plays a crucial role in generalization, beyond the conventional view of merely augmenting systems with specific capabilities.

翻译：暂无翻译