The Gaussian process state-space model (GPSSM) has attracted much attention over the past decade. However, the model representation power of the GPSSM is far from satisfactory. Most GPSSM studies rely on the standard Gaussian process (GP) with a preliminary kernel, such as the squared exponential (SE) kernel or Mat\'{e}rn kernel, which limits the model representation power and its application in complex scenarios. To address this issue, this paper proposes a novel class of probabilistic state-space models, called TGPSSMs. By leveraging a parametric normalizing flow, the TGPSSMs enrich the GP priors in the standard GPSSM, rendering the state-space model more flexible and expressive. Additionally, we present a scalable variational inference algorithm for learning and inference in TGPSSMs, which provides a flexible and optimal structure for the variational distribution of latent states. The algorithm is interpretable and computationally efficient owing to the sparse representation of GP and the bijective nature of normalizing flow. To further improve the learning and inference performance of the proposed algorithm, we integrate a constrained optimization framework to enhance the state-space representation capabilities and optimize the hyperparameters. The experimental results based on various synthetic and real datasets corroborate that the proposed TGPSSM yields superior learning and inference performance compared to several state-of-the-art methods. The accompanying source code is available at \url{https://github.com/zhidilin/TGPSSM}.

翻译：高斯过程状态空间模型（GPSSM）在过去十年中备受关注。然而，GPSSM的模型表示能力远未令人满意。多数GPSSM研究依赖于具有初步核函数的（如平方指数核或马特恩核）标准高斯过程，这限制了模型表示能力及其在复杂场景中的应用。为应对这一问题，本文提出了一类新型概率状态空间模型，即TGPSSM。通过利用参数化归一化流，TGPSSM丰富了标准GPSSM中的高斯过程先验，使状态空间模型更具灵活性和表达力。此外，我们提出了一种可扩展的变分推断算法用于TGPSSM的学习与推理，该算法能为潜在状态的变分分布提供灵活且最优的结构。得益于高斯过程的稀疏表示与归一化流的双射特性，该算法具有可解释性和计算高效性。为进一步提升所提算法的学习与推理性能，我们整合了约束优化框架以增强状态空间表示能力并优化超参数。基于多种合成与真实数据集的实验结果证实，与多项先进方法相比，所提出的TGPSSM在学习和推理性能上均表现更优。相关源代码可从 \url{https://github.com/zhidilin/TGPSSM} 获取。