Large language models (LLMs) are in need of sufficient contexts to handle many critical applications, such as retrieval augmented generation and few-shot learning. However, due to the constrained window size, the LLMs can only access to the information within a limited context. Although the size of context window can be extended by fine-tuning, it will result in a substantial cost in both training and inference stage. In this paper, we present Extensible Tokenization as an alternative method which realizes the flexible scaling of LLMs' context. Extensible Tokenization stands as a midware in between of the tokenized context and the LLM, which transforms the raw token embeddings into the extensible embeddings. Such embeddings provide a more compact representation for the long context, on top of which the LLM is able to perceive more information with the same context window. Extensible Tokenization is also featured by its flexibility: the scaling factor can be flexibly determined within a feasible scope, leading to the extension of an arbitrary context length at the inference time. Besides, Extensible Tokenization is introduced as a drop-in component, which can be seamlessly plugged into not only the LLM itself and but also its fine-tuned derivatives, bringing in the extended contextual information while fully preserving the LLM's existing capabilities. We perform comprehensive experiments on long-context language modeling and understanding tasks, which verify Extensible Tokenization as an effective, efficient, flexible, and compatible method to extend LLM's context. Our model and source code will be made publicly available.

翻译：大语言模型（LLM）在处理检索增强生成和少样本学习等关键应用时，需要足够长的上下文支持。然而受限于窗口长度，LLM只能访问有限范围内的信息。虽然可以通过微调扩展上下文窗口，但这会在训练和推理阶段产生巨大成本。本文提出可扩展分词作为替代方法，实现LLM上下文的灵活缩放。可扩展分词作为介于分词上下文与LLM之间的中间件，将原始词嵌入转换为可扩展嵌入。这种嵌入能为长上下文提供更紧凑的表示，使LLM在相同上下文窗口下能够感知更多信息。可扩展分词还具有灵活性：缩放因子可在可行范围内灵活确定，从而在推理时扩展任意上下文长度。此外，可扩展分词作为即插即用组件，不仅能无缝集成到LLM本身，还可嵌入其微调衍生模型，在完全保留LLM现有能力的同时引入扩展的上下文信息。我们在长上下文语言建模与理解任务上进行了全面实验，验证了可扩展分词作为扩展LLM上下文的有效、高效、灵活且兼容的方法。我们的模型与源代码将公开发布。