Modeling long-term dependencies for audio signals is a particularly challenging problem, as even small-time scales yield on the order of a hundred thousand samples. With the recent advent of Transformers, neural architectures became good at modeling dependencies over longer time scales, but they suffered from quadratic constraints to scale them. We propose a generative auto-regressive architecture that can model audio waveforms over quite a large context, greater than 500,000 samples. Our work is adapted to learn time dependencies by learning a latent representation by a CNN front-end, and then learning dependencies over these representations using Transformer encoders, fully trained end-to-end: thereby allowing to learn representations as it deems fit for the next sample. Unlike previous works that compared different time scales to show improvement, we use a standard dataset, with the same number of parameters/context to show improvements. We achieve a state-of-the-art performance as compared to other approaches such as Wavenet, SaSHMI, and Sample-RNN on a standard dataset for modeling long-term structure. This work gives very exciting direction for the field, given improvements in context modeling that can be scaled with more data, as well as potentially better results by using billions/trillions of parameters.

翻译：对音频信号进行长期依赖建模是一个极具挑战性的问题，因为即使是在微小的时间尺度上也会产生数万个样本。随着Transformer技术的近期兴起，神经架构在更长的时间尺度依赖建模方面表现出色，但其扩展性受限于二次复杂度的约束。我们提出了一种生成式自回归架构，能够在超过50万个样本的较大上下文范围内对音频波形进行建模。我们的方法通过CNN前端学习潜在表示，然后利用Transformer编码器学习这些表示之间的依赖关系，并实现端到端的完整训练——从而能够自适应地学习适合下一个样本的表示。与以往通过比较不同时间尺度来展示改进的工作不同，我们采用标准数据集，在相同参数/上下文规模下展示改进效果。在用于长时结构建模的标准数据集上，我们相较于Wavenet、SaSHMI和Sample-RNN等其他方法取得了最先进的性能。鉴于上下文建模的改进可通过更多数据实现规模化扩展，并有望通过使用数十亿/万亿级参数获得更优结果，本研究为该领域开辟了极具前景的发展方向。