Modern Recurrent Neural Networks (RNNs), such as RWKV, are distinguished by their powerful short-range modeling capabilities and efficient fixed-size states, which constitute a core advantage over standard Transformers. However, there is a significant lack of research into their internal state as an editable knowledge representation. To fill this gap, we first explore the representational properties of the RWKV state by proposing the DREAMSTATE framework. This framework utilizes a conditional Diffusion Transformer (DiT) to directly model the probability manifold of the state, enabling its generation and editing. The structural nature of this representation is validated through t-SNE visualizations and controlled generation experiments. After successfully uncovering and modeling the state's representational potential, we further propose a novel hybrid architecture that combines the local advantages of RNNs with global context adaptability. This architecture features a parallel DiT that processes a variable-length global context to dynamically generate and adjust the core recurrent module's WKV parameters, transforming the fixed recurrence mechanism into a context-aware dynamic function. Experiments demonstrate that this hybrid model can be trained stably via a multi-objective loss, validating its design feasibility. Our work not only opens a new research direction for RNN state representation but also provides a concrete architectural reference for future model design. The code is publicly available at: https://huggingface.co/2dgx41s/DreamState.

翻译：现代循环神经网络（RNN），如RWKV，以其强大的短程建模能力和高效的固定大小状态而著称，这构成了其相对于标准Transformer的核心优势。然而，对其内部状态作为一种可编辑知识表示的研究存在显著不足。为填补这一空白，我们首先通过提出DREAMSTATE框架来探索RWKV状态的表示特性。该框架利用条件扩散Transformer（DiT）直接对状态的概率流形进行建模，从而实现其生成与编辑。通过t-SNE可视化和受控生成实验，验证了该表示的结构性本质。在成功揭示并建模状态的表示潜力后，我们进一步提出了一种新颖的混合架构，该架构结合了RNN的局部优势与全局上下文适应性。此架构包含一个并行DiT，用于处理可变长度的全局上下文，以动态生成和调整核心循环模块的WKV参数，从而将固定循环机制转变为上下文感知的动态函数。实验表明，该混合模型可以通过多目标损失函数稳定训练，验证了其设计可行性。我们的工作不仅为RNN状态表示开辟了新的研究方向，也为未来模型设计提供了具体的架构参考。代码公开于：https://huggingface.co/2dgx41s/DreamState。