Large language models (LLMs) have achieved remarkable progress across diverse tasks, yet their internal mechanisms remain largely opaque. In this work, we investigate a fundamental question: to what extent can the original input text be recovered from a single last-token representation in an LLM? To this end, we propose Rep2Text, a novel framework for decoding text from last-token representations. Rep2Text employs a trainable adapter that maps a target model's last-token representation into the token embedding space of a decoding language model, which then autoregressively reconstructs the input text. Experiments across various model combinations (Llama-3.1-8B, Gemma-7B, Mistral-7B-v0.1, Llama-3.2-3B, etc.) show that, on average, roughly half of the tokens in 16-token sequences can be recovered from this compressed representation while preserving strong semantic coherence. Further analysis reveals a clear information bottleneck effect: as sequence length increases, token-level recovery declines, while semantic information remains relatively well preserved. We also find that scaling effects are less pronounced in inversion tasks. Finally, our framework demonstrates robust generalization to out-of-distribution clinical data.

翻译：大语言模型（LLMs）在各类任务中取得了显著进展，但其内部机制仍大多不透明。本研究探讨一个基础性问题：从单个最后令牌表示中，能在多大程度上恢复原始输入文本？为此，我们提出Rep2Text——一种从最后令牌表示中解码文本的新框架。Rep2Text采用可训练适配器，将目标模型的最后令牌表示映射至解码语言模型的令牌嵌入空间，随后该模型通过自回归方式重建输入文本。在多种模型组合（Llama-3.1-8B、Gemma-7B、Mistral-7B-v0.1、Llama-3.2-3B等）上的实验表明，平均而言，16个令牌序列中约半数令牌可从这种压缩表示中恢复，同时保持较强的语义连贯性。进一步分析揭示了清晰的信息瓶颈效应：随着序列长度增加，令牌级恢复率下降，而语义信息相对保持较好。我们还发现，缩放效应在反转任务中表现较弱。最后，该框架对分布外临床数据展现出稳健的泛化能力。