Unified models (UMs) hold promise for their ability to understand and generate content across heterogeneous modalities. Compared to merely generating visual content, the use of UMs for interleaved cross-modal reasoning is more promising and valuable, e.g., for solving understanding problems that require dense visual thinking, improving visual generation through self-reflection, or modeling visual dynamics of the physical world guided by stepwise action interventions. However, existing UMs necessitate pixel decoding as a bridge due to their disjoint visual representations for understanding and generation, which is both ineffective and inefficient. In this paper, we introduce LatentUM, a novel unified model that represents all modalities within a shared semantic latent space, eliminating the need for pixel-space mediation between visual understanding and generation. This design naturally enables flexible interleaved cross-modal reasoning and generation. Beyond improved computational efficiency, the shared representation substantially alleviates codec bias and strengthens cross-modal alignment, allowing LatentUM to achieve state-of-the-art performance on the Visual Spatial Planning benchmark, push the limits of visual generation through self-reflection, and support world modeling by predicting future visual states within the shared semantic latent space.

翻译：统一模型（UMs）因能够理解和生成跨异构模态的内容而展现出潜力。与单纯生成视觉内容相比，利用UMs进行交错式跨模态推理更具前景和价值，例如：解决需密集视觉思维的复杂理解问题、通过自我反思改进视觉生成，或在逐步动作干预指导下对物理世界的视觉动态进行建模。然而，现有UMs因理解和生成任务采用不连通的视觉表征而需借助像素解码作为桥梁，导致模型既低效又无效。本文提出LatentUM——一种新型统一模型，将所有模态表征于共享的语义潜在空间中，消除了视觉理解与生成之间对像素空间中介的依赖。该设计自然支持灵活的交错式跨模态推理与生成。除提升计算效率外，共享表征显著缓解了编解码偏差并强化了跨模态对齐，使LatentUM在视觉空间规划基准上取得最优性能，通过自我反思突破视觉生成极限，并在共享语义潜在空间中预测未来视觉状态以支持世界建模。