The performance of Large Language Models (LLMs) on natural language tasks can be improved through both supervised fine-tuning (SFT) and in-context learning (ICL), which operate via distinct mechanisms. Supervised fine-tuning updates the model's weights by minimizing loss on training data, whereas in-context learning leverages task demonstrations embedded in the prompt, without changing the model's parameters. This study investigates the effects of these learning paradigms on the hidden representations of LLMs using Intrinsic Dimension (ID). We use ID to estimate the number of degrees of freedom between representations extracted from LLMs as they perform specific natural language tasks. We first explore how the ID of LLM representations evolves during SFT and how it varies due to the number of demonstrations in ICL. We then compare the IDs induced by SFT and ICL and find that ICL consistently induces a higher ID compared to SFT, suggesting that representations generated during ICL reside in higher dimensional manifolds in the embedding space.

翻译：大语言模型（LLMs）在自然语言任务上的性能可通过监督微调（SFT）和上下文学习（ICL）两种机制不同的方式提升。监督微调通过最小化训练数据上的损失来更新模型权重，而上下文学习则利用嵌入在提示中的任务示例，不改变模型参数。本研究利用内在维度（ID）探究这两种学习范式对LLMs隐藏表示的影响。我们使用ID来估计LLMs执行特定自然语言任务时提取的表示之间的自由度数量。首先，我们探究了LLM表示的内在维度在SFT过程中的演变规律，以及其在ICL中因示例数量不同而产生的变化。随后，我们比较了SFT与ICL所诱导的内在维度，发现ICL始终诱导出比SFT更高的内在维度，这表明ICL过程中生成的表示位于嵌入空间中更高维的流形上。