Code World Models (CWMs) are language models trained to simulate program execution by predicting explicit runtime state after every executed command. This execution-based world modeling enables internal verification within the model, offering an alternative to natural language chain-of-thought reasoning. However, the sources of errors and the nature of CWMs' limitations remain poorly understood. We study CWMs from two complementary perspectives: local semantic execution and long-horizon state tracking. On real-code benchmarks, we identify two dominant failure regimes. First, dense runtime state reveals produce token-intensive execution traces, leading to token-budget exhaustion on programs with long execution histories. Second, failures disproportionately concentrate in string-valued state, which we attribute to limitations of subword tokenization rather than program structure. To study long-horizon behavior, we use a controlled permutation-tracking benchmark that isolates state propagation under action execution. We show that long-horizon degradation is driven primarily by incorrect action generation: when actions are replaced with ground-truth commands, a Transformer-based CWM propagates state accurately over long horizons, despite known limitations of Transformers in long-horizon state tracking. These findings suggest directions for more efficient supervision and state representations in CWMs that are better aligned with program execution and data types.

翻译：代码世界模型（CWMs）是通过预测每条指令执行后的显式运行时状态来训练、以模拟程序执行的语言模型。这种基于执行的世界建模使模型能够进行内部验证，为自然语言链式推理提供了替代方案。然而，其错误来源与局限性本质仍未得到充分理解。我们从两个互补视角研究CWMs：局部语义执行与长时程状态追踪。在真实代码基准测试中，我们识别出两种主要的失效机制。首先，密集运行时状态表示会生成令牌密集型的执行轨迹，导致具有长执行历史的程序出现令牌预算耗尽。其次，失效异常集中于字符串值状态，我们将其归因于子词令牌化的局限性而非程序结构。为研究长时程行为，我们使用受控的排列追踪基准测试，以隔离动作执行下的状态传播。研究表明，长时程性能下降主要由错误动作生成驱动：当动作被替换为真实指令时，基于Transformer的CWM能够在长时程中准确传播状态，尽管已知Transformer在长时程状态追踪中存在局限。这些发现为CWMs中更高效的监督机制和状态表示指明了方向，使其能更好地与程序执行及数据类型对齐。