The Gaussian process state-space model (GPSSM) has garnered considerable attention over the past decade. However, the standard GP with a preliminary kernel, such as the squared exponential kernel or Mat\'{e}rn kernel, that is commonly used in GPSSM studies, limits the model's representation power and substantially restricts its applicability to complex scenarios. To address this issue, we propose a new class of probabilistic state-space models called TGPSSMs, which leverage a parametric normalizing flow to enrich the GP priors in the standard GPSSM, enabling greater flexibility and expressivity. Additionally, we present a scalable variational inference algorithm that offers a flexible and optimal structure for the variational distribution of latent states. The proposed algorithm is interpretable and computationally efficient due to the sparse GP representation and the bijective nature of normalizing flow. Moreover, we incorporate a constrained optimization framework into the algorithm to enhance the state-space representation capabilities and optimize the hyperparameters, leading to superior learning and inference performance. Experimental results on synthetic and real datasets corroborate that the proposed TGPSSM outperforms several state-of-the-art methods. The accompanying source code is available at \url{https://github.com/zhidilin/TGPSSM}.

翻译：高斯过程状态空间模型在过去十年中引起了广泛关注。然而，标准GPSSM研究中常用的带有初步核函数——如平方指数核或Matern核——的标准高斯过程，限制了模型的表示能力，并大幅削弱了其在复杂场景中的适用性。为解决这一问题，我们提出了一类新型概率状态空间模型，即TGPSSM，该模型利用参数化归一化流来丰富标准GPSSM中的高斯过程先验，从而增强灵活性和表达能力。此外，我们提出了一种可扩展的变分推理算法，该算法为潜在状态的变分分布提供了灵活且最优的结构。由于稀疏高斯过程表示和归一化流的双射特性，所提算法具有可解释性和计算高效性。同时，我们将约束优化框架融入该算法，以提升状态空间表示能力并优化超参数，从而实现更优的学习和推理性能。在合成数据和真实数据集上的实验结果表明，所提TGPSSM模型优于多种最先进方法。配套源代码可在\url{https://github.com/zhidilin/TGPSSM}获取。