One way of introducing sparsity into deep networks is by attaching an external table of parameters that is sparsely looked up at different layers of the network. By storing the bulk of the parameters in the external table, one can increase the capacity of the model without necessarily increasing the inference time. Two crucial questions in this setting are then: what is the lookup function for accessing the table and how are the contents of the table consumed? Prominent methods for accessing the table include 1) using words/wordpieces token-ids as table indices, 2) LSH hashing the token vector in each layer into a table of buckets, and 3) learnable softmax style routing to a table entry. The ways to consume the contents include adding/concatenating to input representation, and using the contents as expert networks that specialize to different inputs. In this work, we conduct rigorous experimental evaluations of existing ideas and their combinations. We also introduce a new method, alternating updates, that enables access to an increased token dimension without increasing the computation time, and demonstrate its effectiveness in language modeling.

翻译：在深层网络中引入稀疏性的一种方法是附加一个外部参数表，并在网络的不同层对其进行稀疏查找。通过将大部分参数存储于外部表中，可以在不必然增加推理时间的情况下提升模型容量。此设置下的两个关键问题是：访问该表的查找函数是什么？以及表内容如何被使用？访问表的常用方法包括：1）使用单词/词片段的标记ID作为表索引，2）对各层中的令牌向量进行局部敏感哈希（LSH）以映射至桶表，3）采用可学习的类Softmax路由至表条目。使用表内容的方式包括：将其添加或拼接至输入表示，以及将其用作针对不同输入的特化专家网络。在本工作中，我们对现有方法及其组合进行了严格的实验评估，并提出了一种名为“交替更新”的新方法，该方法能够在不增加计算时间的情况下实现对更高令牌维度的访问，并在语言建模任务中验证了其有效性。