Generative sequence models are typically trained on sample sequences from natural or formal languages. It is a crucial question whether -- or to what extent -- sample-based training is able to capture the true structure of these languages, often referred to as the ``world model''. Theoretical results indicate that we can hope for soundness at best, that is, generating valid sequences, but not necessarily all of them. However, it is still important to have practical tools that are able to verify whether a given sequence model is sound. In this study, we focus on chess, as it is a domain that provides enough complexity while having a simple rule-based world model. We propose adversarial sequence generation for verifying the soundness of the sequence model. Our adversaries generate valid sequences so as to force the sequence model to generate an invalid next move prediction. Apart from the falsification of soundness, this method is also suitable for a more fine-grained analysis of the failure modes and the effects of different choices during training. To demonstrate this, we propose a number of methods for adversarial sequence generation and evaluate the approach on a large set of chess models. We train models on random as well as high-quality chess games, using several training recipes. We find that none of the models are sound, but some training techniques and dataset choices are able to improve soundness remarkably. We also investigate the potential application of board state probes in both our training and attack methods. Our findings indicate that the extracted board states have no causal role in next token prediction in most of the models.

翻译：生成式序列模型通常基于自然语言或形式语言的样本序列进行训练。一个关键问题是：基于样本的训练能否（或在多大程度上）捕捉到这些语言的真实结构，即通常所称的“世界模型”。理论研究表明，我们最多只能期望模型具备可靠性，即生成有效的序列，但不一定能生成所有有效序列。然而，开发能够验证给定序列模型是否可靠的实际工具仍然至关重要。本研究以国际象棋为研究对象，因其领域复杂度适中且具备基于规则的简单世界模型。我们提出通过对抗序列生成来验证序列模型的可靠性。我们的对抗方法通过生成有效序列，迫使序列模型产生无效的下一步预测。除了证伪可靠性之外，该方法还适用于对故障模式及训练过程中不同选择的影响进行更细粒度的分析。为验证此方法，我们提出了多种对抗序列生成策略，并在大量国际象棋模型上进行了评估。我们使用随机棋局和高质量棋局数据集，采用多种训练方案训练模型。研究发现：所有模型均不具备完全可靠性，但某些训练技术和数据集选择能显著提升可靠性。我们还探究了棋盘状态探针在训练和攻击方法中的潜在应用。结果表明，在大多数模型中，提取的棋盘状态对下一标记预测不具有因果作用。