With the development of transformer-based large language models (LLMs), they have been applied to many fields due to their remarkable utility, but this comes at a considerable computational cost at deployment. Fortunately, some methods such as pruning or constructing a mixture of experts (MoE) aim at exploiting sparsity in transformer feedforward (FF) blocks to gain boosts in speed and reduction in memory requirements. However, these techniques can be very costly and inflexible in practice, as they often require training or are restricted to specific types of architectures. To address this, we introduce GRIFFIN, a novel training-free MoE that selects unique FF experts at the sequence level for efficient generation across a plethora of LLMs with different non-ReLU activation functions. This is possible due to a critical observation that many trained LLMs naturally produce highly structured FF activation patterns within a sequence, which we call flocking. Despite our method's simplicity, we show with 50% of the FF parameters, GRIFFIN maintains the original model's performance with little to no degradation on a variety of classification and generation tasks, all while improving latency (e.g. 1.25$\times$ speed-up in Llama 2 13B on an NVIDIA L40). Code is available at https://github.com/hdong920/GRIFFIN.

翻译：随着基于Transformer的大语言模型（LLMs）的发展，因其卓越的实用性已被广泛应用于众多领域，但这在部署过程中带来了巨大的计算成本。幸运的是，某些方法（如剪枝或构建混合专家模型（MoE））旨在利用Transformer前馈（FF）块中的稀疏性，以提升速度并降低内存需求。然而，这些技术在实践中可能成本高昂且缺乏灵活性，因为它们通常需要训练或受限于特定类型的架构。为解决这一问题，我们提出了GRIFFIN——一种无需训练的新型MoE，它能在序列级别选择独特的FF专家，从而在具有不同非ReLU激活函数的多种LLMs中实现高效生成。这得益于一个关键发现：许多已训练的LLMs在序列内自然地产生高度结构化的FF激活模式，我们称之为"聚集效应"。尽管方法简单，但我们证明，在仅保留50%的FF参数情况下，GRIFFIN能在多种分类和生成任务上保持原始模型的性能，几乎无性能下降，同时提升延迟效率（例如，在NVIDIA L40上对Llama 2 13B实现1.25倍加速）。代码已开源：https://github.com/hdong920/GRIFFIN。