Toeplitz Neural Networks (TNNs) (Qin et. al. 2023) are a recent sequence model with impressive results. They require O(n log n) computational complexity and O(n) relative positional encoder (RPE) multi-layer perceptron (MLP) and decay bias calls. We aim to reduce both. We first note that the RPE is a non-SPD (symmetric positive definite) kernel and the Toeplitz matrices are pseudo-Gram matrices. Further 1) the learned kernels display spiky behavior near the main diagonals with otherwise smooth behavior; 2) the RPE MLP is slow. For bidirectional models, this motivates a sparse plus low-rank Toeplitz matrix decomposition. For the sparse component's action, we do a small 1D convolution. For the low rank component, we replace the RPE MLP with linear interpolation and use asymmetric Structured Kernel Interpolation (SKI) (Wilson et. al. 2015) for O(n) complexity: we provide rigorous error analysis. For causal models, "fast" causal masking (Katharopoulos et. al. 2020) negates SKI's benefits. Working in the frequency domain, we avoid an explicit decay bias. To enforce causality, we represent the kernel via the real part of its frequency response using the RPE and compute the imaginary part via a Hilbert transform. This maintains O(n log n) complexity but achieves an absolute speedup. Modeling the frequency response directly is also competitive for bidirectional training, using one fewer FFT. We set a speed state of the art on Long Range Arena (Tay et. al. 2020) with minimal score degradation.

翻译：Toeplitz神经网络（TNNs）（Qin等人，2023）是近期提出的一种序列模型，性能出色。该模型需要O(n log n)的计算复杂度，以及O(n)次相对位置编码（RPE）多层感知器（MLP）和衰减偏置调用。我们旨在同时降低这两方面开销。首先，我们注意到RPE是非对称正定（non-SPD）核，且Toeplitz矩阵是伪Gram矩阵。此外：1）学习到的核在主对角线附近呈现尖峰行为，其余区域则平滑；2）RPE MLP速度较慢。对于双向模型，这启发我们采用稀疏加低秩的Toeplitz矩阵分解。稀疏分量部分通过小型一维卷积实现，低秩分量部分则用线性插值替代RPE MLP，并利用非对称结构化核插值（SKI）（Wilson等人，2015）实现O(n)复杂度，同时提供严格的误差分析。对于因果模型，“快速”因果掩码（Katharopoulos等人，2020）抵消了SKI的优势。在频域处理时，我们避免显式使用衰减偏置。为强制因果关系，我们通过RPE表示核的频响实部，并借助希尔伯特变换计算虚部。该方法保持O(n log n)复杂度，但实现了绝对加速。直接建模频响在双向训练中也具有竞争力，可减少一次FFT运算。我们在Long Range Arena基准（Tay等人，2020）上以极小的分数损失创下速度新纪录。