Large Language Models (LLMs) excel at problem solving by generating chain of thoughts in natural language, but such verbal thinking is computationally costly and prone to overthinking. A recent work instead proposes a latent thinking architecture, Huginn-3.5B, which represents intermediate reasoning steps as a sequence of latent representations. However, latent thoughts lack interpretability and are difficult to supervise, raising concerns about the correctness and reliability of the model's latent thinking processes. In this paper, we provide a systematic study of how Huginn-3.5B thinks in the latent space and how external supervision signals can improve its latent thinking processes. We show that latent thoughts leading to correct versus incorrect answers exhibit highly distinguishable patterns, and that a latent classifier can reliably predict answer correctness directly from latent thoughts. Leveraging these insights, we propose Latent Thinking Optimization (LTO), a probabilistic algorithm that employs the latent classifier as a Latent Reward Model (LRM) to optimize the latent thinking processes. Extensive experiments across diverse reasoning tasks demonstrate that LRM is highly effective in detecting incorrect latent thinking patterns, and LTO can significantly improve the latent thinking processes. Furthermore, we show that LRM can generalize across diverse domains, and LTO can be seamlessly applied to general LLMs to improve their thinking processes. In contrast to verbal thinking, our method demonstrates that reward modeling and scaling test-time thinking with supervision can be performed directly in the latent space, highlighting its potential as a general, efficient, and domain-agnostic approach to improving the thinking processes of LLMs.

翻译：大型语言模型（LLM）通过生成自然语言的思维链在问题解决方面表现出色，但这种言语式思维计算成本高昂且容易陷入过度思考。近期一项研究提出了一种潜在思维架构Huginn-3.5B，它将中间推理步骤表示为一系列潜在表征。然而，潜在思维缺乏可解释性且难以监督，引发了关于模型潜在思维过程正确性与可靠性的担忧。本文系统研究了Huginn-3.5B如何在潜在空间中进行思考，以及外部监督信号如何改进其潜在思维过程。我们证明导致正确与错误答案的潜在思维呈现出高度可区分的模式，且潜在分类器能够直接从潜在思维可靠预测答案正确性。基于这些发现，我们提出潜在思维优化（LTO）——一种概率算法，该算法利用潜在分类器作为潜在奖励模型（LRM）来优化潜在思维过程。跨多样化推理任务的大量实验表明，LRM在检测错误潜在思维模式方面非常有效，且LTO能显著改进潜在思维过程。此外，我们证明LRM能够泛化至不同领域，且LTO可无缝应用于通用LLM以改进其思维过程。与言语式思维相比，我们的方法表明奖励建模及带监督的测试时思维扩展可直接在潜在空间中实现，这凸显了其作为一种通用、高效且领域无关的改进LLM思维方法的潜力。