Large Language Models (LLMs) are unable to reliably reason about specific physical systems. Attempts to imbue LLMs with knowledge of the necessary physics concepts have shown great promise, but explainability and validation remain open challenges. An emerging alternative is tooling, where LLMs can query physical simulators and use the resulting simulation traces as context for validation. This approach suffers from poor scalability since simulation traces contain large volumes of fine-grained numerical and semantic data. We show that translating simulation traces to a sparse representation of "high-level" structural patterns leads to more effective interpretation by LLMs. We propose an unsupervised learning scheme to perform this translation, or annotation, via program synthesis. Our learning results in a library of programs that act as pattern detectors which can translate simulation traces to sparse, annotated pattern sequences. The detected patterns may optionally be guided by human experts via string labels (rigid collision, stretching spring, etc.). We show, using a recent physics benchmark, that such annotated representations are more amenable to natural language reasoning about specific physical systems. The synthesized programs serve as transparent, explainable functions that map system states to a sparse and efficient annotation space. As an example application, we show how goals within physical systems that are specified in natural language may be converted to reward programs which are maximized to find solutions.

翻译：大型语言模型（LLMs）无法可靠地推理特定物理系统。尽管向LLMs注入必要物理概念知识的尝试已展现出巨大潜力，但其可解释性与验证仍面临挑战。一种新兴替代方案是工具化方法：LLMs可查询物理仿真器，并利用生成的仿真轨迹作为验证上下文。然而，由于仿真轨迹包含大量细粒度数值与语义数据，该方法的可扩展性较差。研究表明，将仿真轨迹转化为“高层”结构模式的稀疏表示，能有效提升LLMs的推理效能。我们提出了一种无监督学习方案，通过程序合成实现这种转化（即标注过程）。该学习结果生成一个程序库，这些程序作为模式检测器，可将仿真轨迹转化为稀疏的、带标注的模式序列。检测到的模式可依据人类专家的字符串标签（如刚性碰撞、弹性弹簧等）进行引导。基于近期物理基准测试，我们证明这种标注表示更有利于针对特定物理系统开展自然语言推理。合成的程序作为透明可解释的函数，将系统状态映射至稀疏高效的标注空间。作为示例应用，我们展示了如何将自然语言指定的物理系统目标转化为奖励程序，通过最大化该程序来寻找解决方案。