Forecasting multivariate hidden Markov processes is challenging due to nonlinear and nonstationary observations, latent state transitions, and cross-sequence dependencies. While deep learning methods achieve strong predictive accuracy, they typically lack explicit state modeling, whereas Hidden Markov Models (HMMs) provide interpretable latent states but struggle with complex nonlinear emissions and scalability. To address these limitations, we propose DRL-STAF, a Deep Reinforcement Learning based STate-Aware Forecasting framework that jointly predicts next-step observations and estimates the corresponding hidden states for complex multivariate hidden Markov processes. Specifically, DRL-STAF models complex nonlinear emissions using deep neural networks and estimates discrete hidden states using reinforcement learning, reducing the reliance on predefined transition structures and enabling flexible adaptation to diverse temporal dynamics. In particular, DRL-STAF mitigates the state-space explosion encountered by typical multivariate HMM-based methods. Extensive experiments demonstrate that DRL-STAF outperforms HMM variants, standalone deep learning models, and existing DL-HMM hybrids in most cases, while also providing reliable hidden-state estimates.

翻译：由于观测数据的非线性和非平稳性、潜在状态转移以及跨序列依赖关系，多变量隐马尔可夫过程的预测极具挑战性。深度学习方法虽能实现强预测精度，但通常缺乏显式状态建模；而隐马尔可夫模型（HMM）虽提供可解释的潜在状态，却难以处理复杂的非线性发射模式且可扩展性差。为应对这些局限，我们提出DRL-STAF——一种基于深度强化学习的状态感知预测框架，能够面向复杂多变量隐马尔可夫过程联合预测下一步观测并估计对应的隐藏状态。具体而言，DRL-STAF利用深度神经网络建模复杂非线性发射，通过强化学习估计离散隐藏状态，从而降低对预定义转移结构的依赖，并灵活适应多样化的时间动态特性。特别地，DRL-STAF缓解了典型多变量HMM方法面临的状态空间爆炸问题。大量实验表明，DRL-STAF在多数情况下优于HMM变体、独立深度学习模型以及现有深度学习与HMM的混合模型，同时能提供可靠的隐藏状态估计。