Large language models (LLMs) predominantly employ decoder-only transformer architectures, necessitating the retention of keys/values information for historical tokens to provide contextual information and avoid redundant computation. However, the substantial size and parameter volume of these LLMs require massive GPU memory. This memory demand increases with the length of the input text, leading to an urgent need for more efficient methods of information storage and processing. This study introduces Anchor-based LLMs (AnLLMs), which utilize an innovative anchor-based self-attention network (AnSAN) and also an anchor-based inference strategy. This approach enables LLMs to compress sequence information into an anchor token, reducing the keys/values cache and enhancing inference efficiency. Experiments on question-answering benchmarks reveal that AnLLMs maintain similar accuracy levels while achieving up to 99% keys/values cache reduction and up to 3.5 times faster inference. Despite a minor compromise in accuracy, the substantial enhancements of AnLLMs employing the AnSAN technique in resource utilization and computational efficiency underscore their potential for practical LLM applications.

翻译：大语言模型主要采用仅解码器变换器架构，需要保留历史标记的键/值信息以提供上下文信息并避免冗余计算。然而，这些大语言模型庞大的规模和参数量需要大量GPU内存。这种内存需求随输入文本长度增加而增长，亟需更高效的信息存储与处理方法。本研究提出了基于锚点的大语言模型（AnLLMs），该模型采用创新的基于锚点的自注意力网络（AnSAN）及基于锚点的推理策略。这种方法使大语言模型能够将序列信息压缩到锚点标记中，减少键/值缓存并提升推理效率。在问答基准测试中，实验表明AnLLMs在保持相似准确率的同时，实现了最高99%的键/值缓存缩减和最高3.5倍的推理加速。尽管准确率略有妥协，但采用AnSAN技术的AnLLMs在资源利用和计算效率方面的显著提升，凸显了其在实用型大语言模型应用中的潜力。