A new breed of gated-linear recurrent neural networks has reached state-of-the-art performance on a range of sequence modeling problems. Such models naturally handle long sequences efficiently, as the cost of processing a new input is independent of sequence length. Here, we explore another advantage of these stateful sequence models, inspired by the success of model merging through parameter interpolation. Building on parallels between fine-tuning and in-context learning, we investigate whether we can treat internal states as task vectors that can be stored, retrieved, and then linearly combined, exploiting the linearity of recurrence. We study this form of fast model merging on Mamba-2.8b, a pretrained recurrent model, and present preliminary evidence that simple linear state interpolation methods suffice to improve next-token perplexity as well as downstream in-context learning task performance.

翻译：一种新型门控线性递归神经网络在多项序列建模问题上已达到最先进性能。此类模型天然具备高效处理长序列的能力，因为处理新输入的计算成本与序列长度无关。受参数插值实现模型合并的成功启发，本文探索了这些带状态序列模型的另一优势。基于微调与上下文学习之间的相似性，我们研究了能否将内部状态视为可存储、检索并线性组合的任务向量，并利用递归的线性特性。我们在预训练递归模型Mamba-2.8b上研究了这种快速模型合并形式，初步实验表明：简单的线性状态插值方法足以改进下一词元困惑度及下游上下文学习任务性能。