We investigate nonlinear prediction/regression in an online setting and introduce a hybrid model that effectively mitigates, via a joint mechanism through a state space formulation, the need for domain-specific feature engineering issues of conventional nonlinear prediction models and achieves an efficient mix of nonlinear and linear components. In particular, we use recursive structures to extract features from raw sequential sequences and a traditional linear time series model to deal with the intricacies of the sequential data, e.g., seasonality, trends. The state-of-the-art ensemble or hybrid models typically train the base models in a disjoint manner, which is not only time consuming but also sub-optimal due to the separation of modeling or independent training. In contrast, as the first time in the literature, we jointly optimize an enhanced recurrent neural network (LSTM) for automatic feature extraction from raw data and an ARMA-family time series model (SARIMAX) for effectively addressing peculiarities associated with time series data. We achieve this by introducing novel state space representations for the base models, which are then combined to provide a full state space representation of the hybrid or the ensemble. Hence, we are able to jointly optimize both models in a single pass via particle filtering, for which we also provide the update equations. The introduced architecture is generic so that one can use other recurrent architectures, e.g., GRUs, traditional time series-specific models, e.g., ETS or other optimization methods, e.g., EKF, UKF. Due to such novel combination and joint optimization, we demonstrate significant improvements in widely publicized real life competition datasets. We also openly share our code for further research and replicability of our results.

翻译：我们研究了在线场景下的非线性预测/回归问题，并提出了一种混合模型。该模型通过状态空间框架下的联合机制，有效缓解了传统非线性预测模型对领域特定特征工程的依赖，并实现了非线性与线性分量的高效融合。具体而言，我们采用递归结构从原始序列中提取特征，同时利用传统线性时间序列模型处理序列数据的复杂特征（如季节性与趋势）。现有最先进的集成或混合模型通常以分离方式训练基模型，不仅耗时，且因建模分离或独立训练而导致性能次优。作为文献中首次尝试，我们提出联合优化增强型循环神经网络（LSTM）以自动提取原始数据特征，以及ARMA族时间序列模型（SARIMAX）以有效处理时间序列数据的特异性。为此，我们为基模型引入新型状态空间表示，并将其融合为混合或集成模型的完整状态空间表示。通过粒子滤波实现单次传递中两个模型的联合优化，并推导了相应的更新方程。所提出的架构具有通用性，可支持其他循环架构（如GRU）、传统时间序列专用模型（如ETS）或其他优化方法（如EKF、UKF）。得益于这种新型组合与联合优化，我们在广泛公开的真实竞赛数据集上取得了显著性能提升。我们同时开源代码以促进后续研究与结果复现。